From 87bda7049f58a05cba7c0e6d53a404e468d6ef14 Mon Sep 17 00:00:00 2001 From: "Daniel Plohmann (bismuth)" Date: Mon, 7 Sep 2026 17:04:44 +0200 Subject: [PATCH 1/6] chore: draft the v4.5.1 release notes The changelog entry for what has landed since v4.5.0 (6cc5aca): #302, #303, #304, #305, #306, #307, #308, #310 and #316, plus the Dependabot ty bump 572acd7 that #302 had to answer for. Held as a draft rather than a release. The version is deliberately NOT bumped here -- #309, #311 and #312 are reviewed and wanted but waiting on a rebase, and splitting one coherent accuracy pass across two releases makes both entries weaker. What this commit does is stop the writeup living in a scratchpad while that happens. To finish it: add sections for #309, #311 and #312, bump VERSION in src/smda/SmdaConfig.py and __version__ in src/smda/__init__.py, and move the date to the actual release day. No escaper output changed anywhere in this set -- the only edit to intel/definitions.py adds two GAP_SEQUENCES entries, which feed padding detection and not escaping, and testEscaperFingerprint passes unchanged -- so ESCAPER_DOWNWARD_COMPATIBILITY stays at 4.4.5 and INTEL_PIC_HASH_ESCAPE_VERSION at 4.3.5, and no report needs reprocessing. Figures are the contributors' own except where this session reproduced them, and the two that were reproduced are stated as measured here: the padding cut's analysis-time cost (+3.4% claimed, +3.84% measured) and the fixture movement on rust_pe_gnu_xored (+26 real starts against +8 false, scored against that fixture's own COFF symbol table rather than the corpus macro mean). Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ec218d57..7aede234 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ Newest first. Entries carry the release date, the version, and what changed. + * 2026-09-07: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the report's address-space contract written down. **Recovery (intel):** a 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of the table read, falling back to the first register-against-immediate compare in the window, and sizing a 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as functions inside the function they belong to, so one defect produced a false positive and a shattered function together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no true positive lost on any cell, 135 of 140 cells bit-identical. The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the previous one ended with, so a function reached by falling through inter-function padding was decoded as a continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between functions with, a fall-through cut additionally requires its seed to decode as a function entry on every format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at +3.4% analysis time (reproduced here at +3.84%). **Recovery (AArch64):** candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset inside the same corpus. Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary code, which cannot separate these, because a case block is preceded by the previous case's terminating branch and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap scan exists to reach. **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into("=0.3.0`, a declared BSD-compatible pure-Python dependency replacing the optional GPL `pdbparse` import whose failure was silent (three-source record merge: procedures, publics, thunks; OMAP-aware RVA resolution; parent-relative fragment attribution `$+0x`; naming 273/4045 to 3698/4045 functions on sqlite3 x64 and 1/477 to 442/477 on Rust x64); MSVC-decorated PDB public records demangle ahead of the Rust evidence gate; PE symbols demangle Itanium names like their ELF/Mach-O siblings and resolve through `section_idx`; demangled Rust names keep rustc spacing instead of Ghidra-style condensation; punycode identifiers decode non-ASCII text; lief symbol names survive invalid UTF-8 via a shared helper; ELF import names stay binary-encoded for import reconstruction and report merging; Mach-O Swift demangling batches through one toolchain call per image (4.29 s to 0.08 s over the 12-sample corpus, identical output); confirmed-format parse failures across providers and loaders warn instead of vanishing at debug. **Probes/loaders:** bitness decided by REX.W usage statistics over code areas; AArch64 recognized in raw buffers; unrecognized inputs pass through to analysis instead of erroring; opt-in `.eh_frame_hdr`/linker-table candidates on both instruction sets; `WITH_STRINGS` no longer destroys a DEX report's string references; a lief type object's repr can no longer reach a DEX report's type field dressed as a type descriptor, since the ` Date: Mon, 7 Sep 2026 17:08:08 +0200 Subject: [PATCH 2/6] chore: fold #318 into the v4.5.1 release notes #318 landed as 395c88d after this entry was written. Its five corrections go under Housekeeping: the README's metadata.language claim, the restored `-> bool`, the ruff-check hook rename, the GAP_SEQUENCES reach note, and the docstring on the queue-rebuild test. Keeping this current as things merge is the point of the branch existing, so it does not go stale between now and the release. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7aede234..2a34ef2c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ Newest first. Entries carry the release date, the version, and what changed. - * 2026-09-07: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the report's address-space contract written down. **Recovery (intel):** a 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of the table read, falling back to the first register-against-immediate compare in the window, and sizing a 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as functions inside the function they belong to, so one defect produced a false positive and a shattered function together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no true positive lost on any cell, 135 of 140 cells bit-identical. The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the previous one ended with, so a function reached by falling through inter-function padding was decoded as a continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between functions with, a fall-through cut additionally requires its seed to decode as a function entry on every format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at +3.4% analysis time (reproduced here at +3.84%). **Recovery (AArch64):** candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset inside the same corpus. Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary code, which cannot separate these, because a case block is preceded by the previous case's terminating branch and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap scan exists to reach. **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into(" bool` back through a `bool()` wrapper rather than keeping the annotation deleted; the ruff hook moved to `id: ruff-check`, astral having renamed it and left `ruff` as a legacy alias; `GAP_SEQUENCES` gained a note that it is read in seven places outside the alignment cut, which is what makes a measurement over a change to that table hard to attribute to one caller; and the queue-rebuild test now says it guards against reintroducing scoring into `addTailcallCandidate`, its assertion having become trivially true when that scoring was removed. No escaper output changed, so `ESCAPER_DOWNWARD_COMPATIBILITY` stays at `4.4.5` and `INTEL_PIC_HASH_ESCAPE_VERSION` at `4.3.5`. **Recovery output moves on binaries these paths touch:** of the bundled fixtures only `rust_pe_gnu_xored` changes, 2,355 to 2,389 function starts; scored against its own COFF symbol table that is +26 real functions and +8 false ones, recall 92.45% to 93.64%. (THX: @r0ny123) * 2026-08-25: v4.5.0 - Function-recovery accuracy pass, MSVC/PDB symbol recovery, synthesis robustness, and the escaper compatibility marker that downstream indexes read. **Recovery (intel/AArch64):** CET `notrack` switch dispatches are now classified as dispatches instead of far jumps and discarded (a single switch previously produced 43 bogus functions); switch tables whose base is register-held are recovered via a bounded `lea` back-walk; jump-table entry scans are bounded against attacker-controlled counts (1049.7 s to 0.002 s) and the address-0 sentinel class is refused at five call sites; the indirect-call backward walk stops resolving against stale registers (branch predecessors, ABI-preserving calls, memory-writing `mov` forms); a function is cut at the CET landing pad it runs into after padding; `push rbp`-opening prologues seed their real entry instead of one byte in; direct calls outrank the inferred alignment floor (recovered `abort` in static glibc). On AArch64, a jump table whose sign-extension is folded into the merging `add` (`add x8, x8, w9, sxtw #2`) is read as signed instead of losing the whole table at its first backward entry, and the constant tracker now drops the registers capstone reports only as implicit writes, so a value held in x30 no longer survives the `bl`/`blr` that clobbers it. On intel, `popa`/`popad` join the instructions known to overwrite eax without naming it, so the syscall-number backtrack stops reporting a `mov eax, N` the pop had already replaced, which is a wrong answer rather than an unresolved one. PE ARM64 exception records are carved from headerless images (0 to 24 functions on veneer-style images) with packed-fragment rejection. Absolute memory operands book data references, which enables a late candidate pass recovering Delphi method-table entries nothing else names (+3 real functions on the corpus, replacing three recoveries that rested on provably wrong stale-register dataflow). Both Delphi method-table walks stop at the first unreadable entry rather than spending a declared 16-bit entry count as an iteration count. The analysis budget is an actual in-pass bound now (polled within passes, latched between them). **Measured against newly bundled labelled ground truth (50 locally built binaries, 11,384 labelled functions): recall 93.781% to 95.098%, body splits 1095 to 306 (-72%), with documented deliberate moves to the `mirai_x64`, `mirai_i386`, `rust_pe_gnu` and `asprox` fixtures.** Median runtime dropped 34% on the 155-file malpedia corpus alongside. **Symbols/demangling:** new vendored MSVC demangler measured per-name against `llvm-undname` on LLVM's own 609-name stress corpus (100% exact spellings, zero third spellings, hostile-input depth and output bounds, shipped with a fuzz target and an attributed reference corpus); `PdbSymbolProvider` rebuilt on `purepdb>=0.3.0`, a declared BSD-compatible pure-Python dependency replacing the optional GPL `pdbparse` import whose failure was silent (three-source record merge: procedures, publics, thunks; OMAP-aware RVA resolution; parent-relative fragment attribution `$+0x`; naming 273/4045 to 3698/4045 functions on sqlite3 x64 and 1/477 to 442/477 on Rust x64); MSVC-decorated PDB public records demangle ahead of the Rust evidence gate; PE symbols demangle Itanium names like their ELF/Mach-O siblings and resolve through `section_idx`; demangled Rust names keep rustc spacing instead of Ghidra-style condensation; punycode identifiers decode non-ASCII text; lief symbol names survive invalid UTF-8 via a shared helper; ELF import names stay binary-encoded for import reconstruction and report merging; Mach-O Swift demangling batches through one toolchain call per image (4.29 s to 0.08 s over the 12-sample corpus, identical output); confirmed-format parse failures across providers and loaders warn instead of vanishing at debug. **Probes/loaders:** bitness decided by REX.W usage statistics over code areas; AArch64 recognized in raw buffers; unrecognized inputs pass through to analysis instead of erroring; opt-in `.eh_frame_hdr`/linker-table candidates on both instruction sets; `WITH_STRINGS` no longer destroys a DEX report's string references; a lief type object's repr can no longer reach a DEX report's type field dressed as a type descriptor, since the ` Date: Tue, 8 Sep 2026 12:44:41 +0200 Subject: [PATCH 3/6] chore: cut v4.5.1 Finishes the draft now that #309, #311 and #312 have landed. Adds their sections, bumps `VERSION` and `__version__` to 4.5.1, and moves the date to the release day. The three new entries: #309's exception-directory refusal of interior gap candidates on x64, with the chained-versus-primary distinction and the seeding-is-not-suppression principle that keeps carved records out of it; #311's four candidate-quality defects, including the corrected entry at `0x40df30` on `aarch64_static_xored` and the metadata-coverage gate; and #312's ARM64 counterpart, which reconstructs the extent an ARM64 `RUNTIME_FUNCTION` does not carry and hoists the shared lookup into `common/`. `USE_MACHO_ADDRESS_REF_CANDIDATES` gets its own **Defaults changed** section. It is the only default whose value moves in this release, it changes recovery output on Mach-O input with no action by the caller, and a default going from off to on inside a follow-ups PR is exactly the kind of thing that gets lost. The closing movement section is rewritten to describe v4.5.0 -> v4.5.1 rather than any single PR, and both figures in it were measured on this tree against independent truth rather than quoted: rust_pe_gnu_xored, vs its 2,186 retained COFF .text function symbols 2,355 -> 2,201 starts, +26 real / -180 false recall 92.452% -> 93.641% precision 85.817% -> 93.003% 11-sample ARM64 Mach-O corpus, vs LC_FUNCTION_STARTS (2,056 truth) TP 1,787 -> 1,818, FP 1,113 -> 1,107 recall 86.916% -> 88.424%, no sample losing recall Two headline figures are marked as the contributor's own because they cannot be reproduced here at all: the `bti j` 803/0 split, since no bundled fixture holds a single `bti j` word, and #312's ARM64 PE result, since a scan of all 104 fixture files finds 8 i386 PEs, 4 AMD64, one ReadyToRun image and no `0xAA64`. Saying which figures are ours and which are theirs is more useful than a uniform tone of confidence. `Turtle_5f9cd91d8d1d`'s 12 dropped starts are attributed rather than left as an unexplained delta: they bisect to #307's inbound-call fix and fall inside the 175 false positives that change already measured, and the sample carries no LC_FUNCTION_STARTS to score them against individually. Closes #319 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr --- CHANGELOG.md | 2 +- src/smda/SmdaConfig.py | 2 +- src/smda/__init__.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a34ef2c..5893fbbf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ Newest first. Entries carry the release date, the version, and what changed. - * 2026-09-07: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the report's address-space contract written down. **Recovery (intel):** a 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of the table read, falling back to the first register-against-immediate compare in the window, and sizing a 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as functions inside the function they belong to, so one defect produced a false positive and a shattered function together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no true positive lost on any cell, 135 of 140 cells bit-identical. The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the previous one ended with, so a function reached by falling through inter-function padding was decoded as a continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between functions with, a fall-through cut additionally requires its seed to decode as a function entry on every format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at +3.4% analysis time (reproduced here at +3.84%). **Recovery (AArch64):** candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset inside the same corpus. Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary code, which cannot separate these, because a case block is preceded by the previous case's terminating branch and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap scan exists to reach. **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into(" bool` back through a `bool()` wrapper rather than keeping the annotation deleted; the ruff hook moved to `id: ruff-check`, astral having renamed it and left `ruff` as a legacy alias; `GAP_SEQUENCES` gained a note that it is read in seven places outside the alignment cut, which is what makes a measurement over a change to that table hard to attribute to one caller; and the queue-rebuild test now says it guards against reintroducing scoring into `addTailcallCandidate`, its assertion having become trivially true when that scoring was removed. No escaper output changed, so `ESCAPER_DOWNWARD_COMPATIBILITY` stays at `4.4.5` and `INTEL_PIC_HASH_ESCAPE_VERSION` at `4.3.5`. **Recovery output moves on binaries these paths touch:** of the bundled fixtures only `rust_pe_gnu_xored` changes, 2,355 to 2,389 function starts; scored against its own COFF symbol table that is +26 real functions and +8 false ones, recall 92.45% to 93.64%. (THX: @r0ny123) + * 2026-09-08: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the report's address-space contract written down. **Recovery (intel):** a 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of the table read, falling back to the first register-against-immediate compare in the window, and sizing a 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as functions inside the function they belong to, so one defect produced a false positive and a shattered function together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no true positive lost on any cell, 135 of 140 cells bit-identical. The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the previous one ended with, so a function reached by falling through inter-function padding was decoded as a continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between functions with, a fall-through cut additionally requires its seed to decode as a function entry on every format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at +3.4% analysis time (reproduced here at +3.84%). The exception directory now also refuses gap candidates it places inside a function. A 64-bit PE carries one `RUNTIME_FUNCTION` record per function, each naming the extent the unwinder needs, so an address strictly inside one is declared to belong to a routine that starts earlier and cannot itself be an entry; only the gap scan ever reaches these addresses, since every one of them sits in a region no reference and no prologue claimed. Two conditions keep it from overreaching. A chained record (`UNW_FLAG_CHAININFO`) describes a fragment of another function, so its first byte is interior too and suppresses on its own, while a primary record's first byte is the entry and stays bookable; and a primary range only suppresses once the analysis has actually recovered the function it names, because a record whose function never analysed is not evidence about what covers the address. The scan then resumes at the extent's end rather than one byte on, which skips the body the record describes instead of re-testing it. Extents are deliberately not merged -- functions are laid out end-to-start, and merging adjacent records would collapse the text section into a handful of spans in which every address but the first reads as interior. Records carved out of an image with no parseable header seed candidates but contribute no extents, which is the distinction that keeps suppression honest: seeding is cheap and self-correcting, suppression is expensive and silent. `USE_PE_X64_PDATA_INTERIOR_GAPS`, on by default. The `is_pe` guard that came with it also stops *seeding* from a non-PE section merely named `.pdata`, which is a candidate-discovery change wider than the title suggests and inert on every bundled image. **Recovery (AArch64):** candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset inside the same corpus. Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary code, which cannot separate these, because a case block is preceded by the previous case's terminating branch and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap scan exists to reach. A follow-up pass then took four candidate-quality defects the BTI series exposed. The interior-BTI test read the candidate snapshot rather than the live function set, so it judged a landing pad against what discovery had proposed instead of against what analysis had recovered. The hoisted-guard heuristic claimed a compare-and-branch could not be a function's first instruction, which is simply false -- on `aarch64_static_xored` the corrected entry sits at `0x40df30`, four bytes ahead of what was booked, and the inter-function `nop` in front of it settles which of the two is the boundary. A metadata-coverage gate now skips the address-materialization scan on an image whose own metadata already named its functions, guarded by a minimum sample count, because a coverage ratio over four resolved call targets says nothing: across the 13 bundled AArch64 images it fires exactly once, on `Turtle_5f9cd91d8d1d` at 98.3% coverage, with output bit-identical either way, so on bundled data it is a work saving with no accuracy consequence. And the ARM64 counterpart of the x64 interior-gap refusal above lands with it: an ARM64 `RUNTIME_FUNCTION` carries no `EndAddress`, so the extent is reconstructed from bits 2-12 of the packed unwind word or from bits 0-17 of the `.xdata` header it points at, both counting instructions. Once an extent is known the rule is identical, which is why the lookup moved into `common/` rather than being written twice; the x64 path keeps the same list, the same bisect and the same running-maximum walk. `USE_PE_ARM64_PDATA_INTERIOR_GAPS`, on by default. Worth knowing that ARM64 PE seeding over-reports on MSVC-built images -- a separated chunk of a routine gets its own record and sets up its own frame, so the entry-shape filter reads it as an entry -- and that no exact test for it is in reach, since 120 of 145 such chunks continue a record with no `.xdata` handler data to name them. **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into(" bool` back through a `bool()` wrapper rather than keeping the annotation deleted; the ruff hook moved to `id: ruff-check`, astral having renamed it and left `ruff` as a legacy alias; `GAP_SEQUENCES` gained a note that it is read in seven places outside the alignment cut, which is what makes a measurement over a change to that table hard to attribute to one caller; and the queue-rebuild test now says it guards against reintroducing scoring into `addTailcallCandidate`, its assertion having become trivially true when that scoring was removed. No escaper output changed, so `ESCAPER_DOWNWARD_COMPATIBILITY` stays at `4.4.5` and `INTEL_PIC_HASH_ESCAPE_VERSION` at `4.3.5`. **Recovery output moves on binaries these paths touch:** eight of the 21 bundled fixtures these paths reach change between v4.5.0 and this release, and the two that carry independent ground truth were scored against it rather than against the contributing PRs' own corpora. `rust_pe_gnu_xored`, against the 2,186 `.text` function symbols its mingw build retains: 2,355 to 2,201 starts, +26 real against -180 false, recall 92.45% to 93.64% and precision 85.82% to 93.00%. The 11-sample ARM64 Mach-O corpus, against `LC_FUNCTION_STARTS` over 2,056 truth functions: 1,787 to 1,818 true positives and 1,113 to 1,107 false ones, recall 86.92% to 88.42%, with no sample losing recall. `aarch64_static_xored` holds at 278 functions with one start corrected four bytes earlier, and `Turtle_5f9cd91d8d1d` drops 12 starts to the inbound-call fix; it carries no `LC_FUNCTION_STARTS`, so those 12 cannot be scored here, but they are part of that change's measured 175 false positives. The remaining 13 are bit-identical. Two headline figures could **not** be reproduced against bundled data and are the contributors' own: the `bti j` 803/0 split, since no bundled fixture contains a single `bti j` word, and the ARM64 PE interior-gap result, since no bundled fixture is an ARM64 PE at all -- of 104 fixture files, 8 are i386 PEs, 4 are AMD64, one is the ReadyToRun image, and none is `0xAA64`. (THX: @r0ny123) * 2026-08-25: v4.5.0 - Function-recovery accuracy pass, MSVC/PDB symbol recovery, synthesis robustness, and the escaper compatibility marker that downstream indexes read. **Recovery (intel/AArch64):** CET `notrack` switch dispatches are now classified as dispatches instead of far jumps and discarded (a single switch previously produced 43 bogus functions); switch tables whose base is register-held are recovered via a bounded `lea` back-walk; jump-table entry scans are bounded against attacker-controlled counts (1049.7 s to 0.002 s) and the address-0 sentinel class is refused at five call sites; the indirect-call backward walk stops resolving against stale registers (branch predecessors, ABI-preserving calls, memory-writing `mov` forms); a function is cut at the CET landing pad it runs into after padding; `push rbp`-opening prologues seed their real entry instead of one byte in; direct calls outrank the inferred alignment floor (recovered `abort` in static glibc). On AArch64, a jump table whose sign-extension is folded into the merging `add` (`add x8, x8, w9, sxtw #2`) is read as signed instead of losing the whole table at its first backward entry, and the constant tracker now drops the registers capstone reports only as implicit writes, so a value held in x30 no longer survives the `bl`/`blr` that clobbers it. On intel, `popa`/`popad` join the instructions known to overwrite eax without naming it, so the syscall-number backtrack stops reporting a `mov eax, N` the pop had already replaced, which is a wrong answer rather than an unresolved one. PE ARM64 exception records are carved from headerless images (0 to 24 functions on veneer-style images) with packed-fragment rejection. Absolute memory operands book data references, which enables a late candidate pass recovering Delphi method-table entries nothing else names (+3 real functions on the corpus, replacing three recoveries that rested on provably wrong stale-register dataflow). Both Delphi method-table walks stop at the first unreadable entry rather than spending a declared 16-bit entry count as an iteration count. The analysis budget is an actual in-pass bound now (polled within passes, latched between them). **Measured against newly bundled labelled ground truth (50 locally built binaries, 11,384 labelled functions): recall 93.781% to 95.098%, body splits 1095 to 306 (-72%), with documented deliberate moves to the `mirai_x64`, `mirai_i386`, `rust_pe_gnu` and `asprox` fixtures.** Median runtime dropped 34% on the 155-file malpedia corpus alongside. **Symbols/demangling:** new vendored MSVC demangler measured per-name against `llvm-undname` on LLVM's own 609-name stress corpus (100% exact spellings, zero third spellings, hostile-input depth and output bounds, shipped with a fuzz target and an attributed reference corpus); `PdbSymbolProvider` rebuilt on `purepdb>=0.3.0`, a declared BSD-compatible pure-Python dependency replacing the optional GPL `pdbparse` import whose failure was silent (three-source record merge: procedures, publics, thunks; OMAP-aware RVA resolution; parent-relative fragment attribution `$+0x`; naming 273/4045 to 3698/4045 functions on sqlite3 x64 and 1/477 to 442/477 on Rust x64); MSVC-decorated PDB public records demangle ahead of the Rust evidence gate; PE symbols demangle Itanium names like their ELF/Mach-O siblings and resolve through `section_idx`; demangled Rust names keep rustc spacing instead of Ghidra-style condensation; punycode identifiers decode non-ASCII text; lief symbol names survive invalid UTF-8 via a shared helper; ELF import names stay binary-encoded for import reconstruction and report merging; Mach-O Swift demangling batches through one toolchain call per image (4.29 s to 0.08 s over the 12-sample corpus, identical output); confirmed-format parse failures across providers and loaders warn instead of vanishing at debug. **Probes/loaders:** bitness decided by REX.W usage statistics over code areas; AArch64 recognized in raw buffers; unrecognized inputs pass through to analysis instead of erroring; opt-in `.eh_frame_hdr`/linker-table candidates on both instruction sets; `WITH_STRINGS` no longer destroys a DEX report's string references; a lief type object's repr can no longer reach a DEX report's type field dressed as a type descriptor, since the ` Date: Tue, 8 Sep 2026 12:48:15 +0200 Subject: [PATCH 4/6] chore: record binweight's serialized type change in the v4.5.1 entry Left open as a question in yesterday's review and resolved as "yes, it belongs in the entry". #302 moved `SmdaFunction.binweight`'s class default from int `0` to float `0.0` while clearing ty 0.0.74's diagnostics, and `binweight` is serialized through `toDict()`. The blast radius is small but real: the per-block accumulation already adds `float(...)`, so every function with at least one block was a float before this release too. Only a function with no blocks -- a zero-function or error report -- keeps the class default, and that value now writes as `0.0` where it wrote `0`. Verified on both trees rather than reasoned about: `SmdaFunction.binweight` is `0` at 6cc5aca and `0.0` at d111548. MCRIT stores these reports, so a consumer diffing them byte-for-byte will see it even though nothing reads the field as an integer. That is exactly the kind of change that costs someone an afternoon if it is not written down. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5893fbbf..cb4bdf09 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ Newest first. Entries carry the release date, the version, and what changed. - * 2026-09-08: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the report's address-space contract written down. **Recovery (intel):** a 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of the table read, falling back to the first register-against-immediate compare in the window, and sizing a 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as functions inside the function they belong to, so one defect produced a false positive and a shattered function together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no true positive lost on any cell, 135 of 140 cells bit-identical. The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the previous one ended with, so a function reached by falling through inter-function padding was decoded as a continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between functions with, a fall-through cut additionally requires its seed to decode as a function entry on every format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at +3.4% analysis time (reproduced here at +3.84%). The exception directory now also refuses gap candidates it places inside a function. A 64-bit PE carries one `RUNTIME_FUNCTION` record per function, each naming the extent the unwinder needs, so an address strictly inside one is declared to belong to a routine that starts earlier and cannot itself be an entry; only the gap scan ever reaches these addresses, since every one of them sits in a region no reference and no prologue claimed. Two conditions keep it from overreaching. A chained record (`UNW_FLAG_CHAININFO`) describes a fragment of another function, so its first byte is interior too and suppresses on its own, while a primary record's first byte is the entry and stays bookable; and a primary range only suppresses once the analysis has actually recovered the function it names, because a record whose function never analysed is not evidence about what covers the address. The scan then resumes at the extent's end rather than one byte on, which skips the body the record describes instead of re-testing it. Extents are deliberately not merged -- functions are laid out end-to-start, and merging adjacent records would collapse the text section into a handful of spans in which every address but the first reads as interior. Records carved out of an image with no parseable header seed candidates but contribute no extents, which is the distinction that keeps suppression honest: seeding is cheap and self-correcting, suppression is expensive and silent. `USE_PE_X64_PDATA_INTERIOR_GAPS`, on by default. The `is_pe` guard that came with it also stops *seeding* from a non-PE section merely named `.pdata`, which is a candidate-discovery change wider than the title suggests and inert on every bundled image. **Recovery (AArch64):** candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset inside the same corpus. Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary code, which cannot separate these, because a case block is preceded by the previous case's terminating branch and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap scan exists to reach. A follow-up pass then took four candidate-quality defects the BTI series exposed. The interior-BTI test read the candidate snapshot rather than the live function set, so it judged a landing pad against what discovery had proposed instead of against what analysis had recovered. The hoisted-guard heuristic claimed a compare-and-branch could not be a function's first instruction, which is simply false -- on `aarch64_static_xored` the corrected entry sits at `0x40df30`, four bytes ahead of what was booked, and the inter-function `nop` in front of it settles which of the two is the boundary. A metadata-coverage gate now skips the address-materialization scan on an image whose own metadata already named its functions, guarded by a minimum sample count, because a coverage ratio over four resolved call targets says nothing: across the 13 bundled AArch64 images it fires exactly once, on `Turtle_5f9cd91d8d1d` at 98.3% coverage, with output bit-identical either way, so on bundled data it is a work saving with no accuracy consequence. And the ARM64 counterpart of the x64 interior-gap refusal above lands with it: an ARM64 `RUNTIME_FUNCTION` carries no `EndAddress`, so the extent is reconstructed from bits 2-12 of the packed unwind word or from bits 0-17 of the `.xdata` header it points at, both counting instructions. Once an extent is known the rule is identical, which is why the lookup moved into `common/` rather than being written twice; the x64 path keeps the same list, the same bisect and the same running-maximum walk. `USE_PE_ARM64_PDATA_INTERIOR_GAPS`, on by default. Worth knowing that ARM64 PE seeding over-reports on MSVC-built images -- a separated chunk of a routine gets its own record and sets up its own frame, so the entry-shape filter reads it as an entry -- and that no exact test for it is in reach, since 120 of 145 such chunks continue a record with no `.xdata` handler data to name them. **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into(" bool` back through a `bool()` wrapper rather than keeping the annotation deleted; the ruff hook moved to `id: ruff-check`, astral having renamed it and left `ruff` as a legacy alias; `GAP_SEQUENCES` gained a note that it is read in seven places outside the alignment cut, which is what makes a measurement over a change to that table hard to attribute to one caller; and the queue-rebuild test now says it guards against reintroducing scoring into `addTailcallCandidate`, its assertion having become trivially true when that scoring was removed. No escaper output changed, so `ESCAPER_DOWNWARD_COMPATIBILITY` stays at `4.4.5` and `INTEL_PIC_HASH_ESCAPE_VERSION` at `4.3.5`. **Recovery output moves on binaries these paths touch:** eight of the 21 bundled fixtures these paths reach change between v4.5.0 and this release, and the two that carry independent ground truth were scored against it rather than against the contributing PRs' own corpora. `rust_pe_gnu_xored`, against the 2,186 `.text` function symbols its mingw build retains: 2,355 to 2,201 starts, +26 real against -180 false, recall 92.45% to 93.64% and precision 85.82% to 93.00%. The 11-sample ARM64 Mach-O corpus, against `LC_FUNCTION_STARTS` over 2,056 truth functions: 1,787 to 1,818 true positives and 1,113 to 1,107 false ones, recall 86.92% to 88.42%, with no sample losing recall. `aarch64_static_xored` holds at 278 functions with one start corrected four bytes earlier, and `Turtle_5f9cd91d8d1d` drops 12 starts to the inbound-call fix; it carries no `LC_FUNCTION_STARTS`, so those 12 cannot be scored here, but they are part of that change's measured 175 false positives. The remaining 13 are bit-identical. Two headline figures could **not** be reproduced against bundled data and are the contributors' own: the `bti j` 803/0 split, since no bundled fixture contains a single `bti j` word, and the ARM64 PE interior-gap result, since no bundled fixture is an ARM64 PE at all -- of 104 fixture files, 8 are i386 PEs, 4 are AMD64, one is the ReadyToRun image, and none is `0xAA64`. (THX: @r0ny123) + * 2026-09-08: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the report's address-space contract written down. **Recovery (intel):** a 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of the table read, falling back to the first register-against-immediate compare in the window, and sizing a 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as functions inside the function they belong to, so one defect produced a false positive and a shattered function together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no true positive lost on any cell, 135 of 140 cells bit-identical. The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the previous one ended with, so a function reached by falling through inter-function padding was decoded as a continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between functions with, a fall-through cut additionally requires its seed to decode as a function entry on every format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at +3.4% analysis time (reproduced here at +3.84%). The exception directory now also refuses gap candidates it places inside a function. A 64-bit PE carries one `RUNTIME_FUNCTION` record per function, each naming the extent the unwinder needs, so an address strictly inside one is declared to belong to a routine that starts earlier and cannot itself be an entry; only the gap scan ever reaches these addresses, since every one of them sits in a region no reference and no prologue claimed. Two conditions keep it from overreaching. A chained record (`UNW_FLAG_CHAININFO`) describes a fragment of another function, so its first byte is interior too and suppresses on its own, while a primary record's first byte is the entry and stays bookable; and a primary range only suppresses once the analysis has actually recovered the function it names, because a record whose function never analysed is not evidence about what covers the address. The scan then resumes at the extent's end rather than one byte on, which skips the body the record describes instead of re-testing it. Extents are deliberately not merged -- functions are laid out end-to-start, and merging adjacent records would collapse the text section into a handful of spans in which every address but the first reads as interior. Records carved out of an image with no parseable header seed candidates but contribute no extents, which is the distinction that keeps suppression honest: seeding is cheap and self-correcting, suppression is expensive and silent. `USE_PE_X64_PDATA_INTERIOR_GAPS`, on by default. The `is_pe` guard that came with it also stops *seeding* from a non-PE section merely named `.pdata`, which is a candidate-discovery change wider than the title suggests and inert on every bundled image. **Recovery (AArch64):** candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset inside the same corpus. Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary code, which cannot separate these, because a case block is preceded by the previous case's terminating branch and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap scan exists to reach. A follow-up pass then took four candidate-quality defects the BTI series exposed. The interior-BTI test read the candidate snapshot rather than the live function set, so it judged a landing pad against what discovery had proposed instead of against what analysis had recovered. The hoisted-guard heuristic claimed a compare-and-branch could not be a function's first instruction, which is simply false -- on `aarch64_static_xored` the corrected entry sits at `0x40df30`, four bytes ahead of what was booked, and the inter-function `nop` in front of it settles which of the two is the boundary. A metadata-coverage gate now skips the address-materialization scan on an image whose own metadata already named its functions, guarded by a minimum sample count, because a coverage ratio over four resolved call targets says nothing: across the 13 bundled AArch64 images it fires exactly once, on `Turtle_5f9cd91d8d1d` at 98.3% coverage, with output bit-identical either way, so on bundled data it is a work saving with no accuracy consequence. And the ARM64 counterpart of the x64 interior-gap refusal above lands with it: an ARM64 `RUNTIME_FUNCTION` carries no `EndAddress`, so the extent is reconstructed from bits 2-12 of the packed unwind word or from bits 0-17 of the `.xdata` header it points at, both counting instructions. Once an extent is known the rule is identical, which is why the lookup moved into `common/` rather than being written twice; the x64 path keeps the same list, the same bisect and the same running-maximum walk. `USE_PE_ARM64_PDATA_INTERIOR_GAPS`, on by default. Worth knowing that ARM64 PE seeding over-reports on MSVC-built images -- a separated chunk of a routine gets its own record and sets up its own frame, so the entry-shape filter reads it as an entry -- and that no exact test for it is in reach, since 120 of 145 such chunks continue a record with no `.xdata` handler data to name them. **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into(" bool` back through a `bool()` wrapper rather than keeping the annotation deleted; the ruff hook moved to `id: ruff-check`, astral having renamed it and left `ruff` as a legacy alias; `GAP_SEQUENCES` gained a note that it is read in seven places outside the alignment cut, which is what makes a measurement over a change to that table hard to attribute to one caller; and the queue-rebuild test now says it guards against reintroducing scoring into `addTailcallCandidate`, its assertion having become trivially true when that scoring was removed. No escaper output changed, so `ESCAPER_DOWNWARD_COMPATIBILITY` stays at `4.4.5` and `INTEL_PIC_HASH_ESCAPE_VERSION` at `4.3.5`. One serialized value does change shape, in a way worth knowing about downstream: `binweight`'s class default moved from int `0` to float `0.0` while the `ty` errors were being cleared, and `binweight` is written into `toDict()`. Every function with at least one block was already a float, since the per-block accumulation adds `float(...)`, so this is visible only on a function that has no blocks at all -- a zero-function or error report -- where the value now serializes as `0.0` rather than `0`. Nothing reads it as an integer, but a consumer diffing stored reports byte-for-byte will see it. **Recovery output moves on binaries these paths touch:** eight of the 21 bundled fixtures these paths reach change between v4.5.0 and this release, and the two that carry independent ground truth were scored against it rather than against the contributing PRs' own corpora. `rust_pe_gnu_xored`, against the 2,186 `.text` function symbols its mingw build retains: 2,355 to 2,201 starts, +26 real against -180 false, recall 92.45% to 93.64% and precision 85.82% to 93.00%. The 11-sample ARM64 Mach-O corpus, against `LC_FUNCTION_STARTS` over 2,056 truth functions: 1,787 to 1,818 true positives and 1,113 to 1,107 false ones, recall 86.92% to 88.42%, with no sample losing recall. `aarch64_static_xored` holds at 278 functions with one start corrected four bytes earlier, and `Turtle_5f9cd91d8d1d` drops 12 starts to the inbound-call fix; it carries no `LC_FUNCTION_STARTS`, so those 12 cannot be scored here, but they are part of that change's measured 175 false positives. The remaining 13 are bit-identical. Two headline figures could **not** be reproduced against bundled data and are the contributors' own: the `bti j` 803/0 split, since no bundled fixture contains a single `bti j` word, and the ARM64 PE interior-gap result, since no bundled fixture is an ARM64 PE at all -- of 104 fixture files, 8 are i386 PEs, 4 are AMD64, one is the ReadyToRun image, and none is `0xAA64`. (THX: @r0ny123) * 2026-08-25: v4.5.0 - Function-recovery accuracy pass, MSVC/PDB symbol recovery, synthesis robustness, and the escaper compatibility marker that downstream indexes read. **Recovery (intel/AArch64):** CET `notrack` switch dispatches are now classified as dispatches instead of far jumps and discarded (a single switch previously produced 43 bogus functions); switch tables whose base is register-held are recovered via a bounded `lea` back-walk; jump-table entry scans are bounded against attacker-controlled counts (1049.7 s to 0.002 s) and the address-0 sentinel class is refused at five call sites; the indirect-call backward walk stops resolving against stale registers (branch predecessors, ABI-preserving calls, memory-writing `mov` forms); a function is cut at the CET landing pad it runs into after padding; `push rbp`-opening prologues seed their real entry instead of one byte in; direct calls outrank the inferred alignment floor (recovered `abort` in static glibc). On AArch64, a jump table whose sign-extension is folded into the merging `add` (`add x8, x8, w9, sxtw #2`) is read as signed instead of losing the whole table at its first backward entry, and the constant tracker now drops the registers capstone reports only as implicit writes, so a value held in x30 no longer survives the `bl`/`blr` that clobbers it. On intel, `popa`/`popad` join the instructions known to overwrite eax without naming it, so the syscall-number backtrack stops reporting a `mov eax, N` the pop had already replaced, which is a wrong answer rather than an unresolved one. PE ARM64 exception records are carved from headerless images (0 to 24 functions on veneer-style images) with packed-fragment rejection. Absolute memory operands book data references, which enables a late candidate pass recovering Delphi method-table entries nothing else names (+3 real functions on the corpus, replacing three recoveries that rested on provably wrong stale-register dataflow). Both Delphi method-table walks stop at the first unreadable entry rather than spending a declared 16-bit entry count as an iteration count. The analysis budget is an actual in-pass bound now (polled within passes, latched between them). **Measured against newly bundled labelled ground truth (50 locally built binaries, 11,384 labelled functions): recall 93.781% to 95.098%, body splits 1095 to 306 (-72%), with documented deliberate moves to the `mirai_x64`, `mirai_i386`, `rust_pe_gnu` and `asprox` fixtures.** Median runtime dropped 34% on the 155-file malpedia corpus alongside. **Symbols/demangling:** new vendored MSVC demangler measured per-name against `llvm-undname` on LLVM's own 609-name stress corpus (100% exact spellings, zero third spellings, hostile-input depth and output bounds, shipped with a fuzz target and an attributed reference corpus); `PdbSymbolProvider` rebuilt on `purepdb>=0.3.0`, a declared BSD-compatible pure-Python dependency replacing the optional GPL `pdbparse` import whose failure was silent (three-source record merge: procedures, publics, thunks; OMAP-aware RVA resolution; parent-relative fragment attribution `$+0x`; naming 273/4045 to 3698/4045 functions on sqlite3 x64 and 1/477 to 442/477 on Rust x64); MSVC-decorated PDB public records demangle ahead of the Rust evidence gate; PE symbols demangle Itanium names like their ELF/Mach-O siblings and resolve through `section_idx`; demangled Rust names keep rustc spacing instead of Ghidra-style condensation; punycode identifiers decode non-ASCII text; lief symbol names survive invalid UTF-8 via a shared helper; ELF import names stay binary-encoded for import reconstruction and report merging; Mach-O Swift demangling batches through one toolchain call per image (4.29 s to 0.08 s over the 12-sample corpus, identical output); confirmed-format parse failures across providers and loaders warn instead of vanishing at debug. **Probes/loaders:** bitness decided by REX.W usage statistics over code areas; AArch64 recognized in raw buffers; unrecognized inputs pass through to analysis instead of erroring; opt-in `.eh_frame_hdr`/linker-table candidates on both instruction sets; `WITH_STRINGS` no longer destroys a DEX report's string references; a lief type object's repr can no longer reach a DEX report's type field dressed as a type descriptor, since the ` Date: Tue, 8 Sep 2026 13:07:42 +0200 Subject: [PATCH 5/6] chore: fold the two remaining #317 items into the v4.5.1 entry #320 and #321 finish the review pass that #318 started, so the release entry should carry all seven of its corrections rather than five. Both are behaviour-neutral typing work, but the Housekeeping text says what each one actually decided rather than listing them as annotations: `blocks` being typed forces the PIC/OPC hash path to refuse an instruction with no bytes instead of blanking it, and `extract_strings` could not be narrowed until three helpers and three `SmdaReport` attributes were declared first. Behaviour-neutrality is stated with the evidence rather than as a claim -- the corpus benchmark ran on #320 and reports 0 of 155 files differing across 175,946 functions, which is a stronger statement than the bundled fixtures can make. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cb4bdf09..3b16cc8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ Newest first. Entries carry the release date, the version, and what changed. - * 2026-09-08: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the report's address-space contract written down. **Recovery (intel):** a 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of the table read, falling back to the first register-against-immediate compare in the window, and sizing a 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as functions inside the function they belong to, so one defect produced a false positive and a shattered function together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no true positive lost on any cell, 135 of 140 cells bit-identical. The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the previous one ended with, so a function reached by falling through inter-function padding was decoded as a continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between functions with, a fall-through cut additionally requires its seed to decode as a function entry on every format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at +3.4% analysis time (reproduced here at +3.84%). The exception directory now also refuses gap candidates it places inside a function. A 64-bit PE carries one `RUNTIME_FUNCTION` record per function, each naming the extent the unwinder needs, so an address strictly inside one is declared to belong to a routine that starts earlier and cannot itself be an entry; only the gap scan ever reaches these addresses, since every one of them sits in a region no reference and no prologue claimed. Two conditions keep it from overreaching. A chained record (`UNW_FLAG_CHAININFO`) describes a fragment of another function, so its first byte is interior too and suppresses on its own, while a primary record's first byte is the entry and stays bookable; and a primary range only suppresses once the analysis has actually recovered the function it names, because a record whose function never analysed is not evidence about what covers the address. The scan then resumes at the extent's end rather than one byte on, which skips the body the record describes instead of re-testing it. Extents are deliberately not merged -- functions are laid out end-to-start, and merging adjacent records would collapse the text section into a handful of spans in which every address but the first reads as interior. Records carved out of an image with no parseable header seed candidates but contribute no extents, which is the distinction that keeps suppression honest: seeding is cheap and self-correcting, suppression is expensive and silent. `USE_PE_X64_PDATA_INTERIOR_GAPS`, on by default. The `is_pe` guard that came with it also stops *seeding* from a non-PE section merely named `.pdata`, which is a candidate-discovery change wider than the title suggests and inert on every bundled image. **Recovery (AArch64):** candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset inside the same corpus. Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary code, which cannot separate these, because a case block is preceded by the previous case's terminating branch and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap scan exists to reach. A follow-up pass then took four candidate-quality defects the BTI series exposed. The interior-BTI test read the candidate snapshot rather than the live function set, so it judged a landing pad against what discovery had proposed instead of against what analysis had recovered. The hoisted-guard heuristic claimed a compare-and-branch could not be a function's first instruction, which is simply false -- on `aarch64_static_xored` the corrected entry sits at `0x40df30`, four bytes ahead of what was booked, and the inter-function `nop` in front of it settles which of the two is the boundary. A metadata-coverage gate now skips the address-materialization scan on an image whose own metadata already named its functions, guarded by a minimum sample count, because a coverage ratio over four resolved call targets says nothing: across the 13 bundled AArch64 images it fires exactly once, on `Turtle_5f9cd91d8d1d` at 98.3% coverage, with output bit-identical either way, so on bundled data it is a work saving with no accuracy consequence. And the ARM64 counterpart of the x64 interior-gap refusal above lands with it: an ARM64 `RUNTIME_FUNCTION` carries no `EndAddress`, so the extent is reconstructed from bits 2-12 of the packed unwind word or from bits 0-17 of the `.xdata` header it points at, both counting instructions. Once an extent is known the rule is identical, which is why the lookup moved into `common/` rather than being written twice; the x64 path keeps the same list, the same bisect and the same running-maximum walk. `USE_PE_ARM64_PDATA_INTERIOR_GAPS`, on by default. Worth knowing that ARM64 PE seeding over-reports on MSVC-built images -- a separated chunk of a routine gets its own record and sets up its own frame, so the entry-shape filter reads it as an entry -- and that no exact test for it is in reach, since 120 of 145 such chunks continue a record with no `.xdata` handler data to name them. **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into(" bool` back through a `bool()` wrapper rather than keeping the annotation deleted; the ruff hook moved to `id: ruff-check`, astral having renamed it and left `ruff` as a legacy alias; `GAP_SEQUENCES` gained a note that it is read in seven places outside the alignment cut, which is what makes a measurement over a change to that table hard to attribute to one caller; and the queue-rebuild test now says it guards against reintroducing scoring into `addTailcallCandidate`, its assertion having become trivially true when that scoring was removed. No escaper output changed, so `ESCAPER_DOWNWARD_COMPATIBILITY` stays at `4.4.5` and `INTEL_PIC_HASH_ESCAPE_VERSION` at `4.3.5`. One serialized value does change shape, in a way worth knowing about downstream: `binweight`'s class default moved from int `0` to float `0.0` while the `ty` errors were being cleared, and `binweight` is written into `toDict()`. Every function with at least one block was already a float, since the per-block accumulation adds `float(...)`, so this is visible only on a function that has no blocks at all -- a zero-function or error report -- where the value now serializes as `0.0` rather than `0`. Nothing reads it as an integer, but a consumer diffing stored reports byte-for-byte will see it. **Recovery output moves on binaries these paths touch:** eight of the 21 bundled fixtures these paths reach change between v4.5.0 and this release, and the two that carry independent ground truth were scored against it rather than against the contributing PRs' own corpora. `rust_pe_gnu_xored`, against the 2,186 `.text` function symbols its mingw build retains: 2,355 to 2,201 starts, +26 real against -180 false, recall 92.45% to 93.64% and precision 85.82% to 93.00%. The 11-sample ARM64 Mach-O corpus, against `LC_FUNCTION_STARTS` over 2,056 truth functions: 1,787 to 1,818 true positives and 1,113 to 1,107 false ones, recall 86.92% to 88.42%, with no sample losing recall. `aarch64_static_xored` holds at 278 functions with one start corrected four bytes earlier, and `Turtle_5f9cd91d8d1d` drops 12 starts to the inbound-call fix; it carries no `LC_FUNCTION_STARTS`, so those 12 cannot be scored here, but they are part of that change's measured 175 false positives. The remaining 13 are bit-identical. Two headline figures could **not** be reproduced against bundled data and are the contributors' own: the `bti j` 803/0 split, since no bundled fixture contains a single `bti j` word, and the ARM64 PE interior-gap result, since no bundled fixture is an ARM64 PE at all -- of 104 fixture files, 8 are i386 PEs, 4 are AMD64, one is the ReadyToRun image, and none is `0xAA64`. (THX: @r0ny123) + * 2026-09-08: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the report's address-space contract written down. **Recovery (intel):** a 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of the table read, falling back to the first register-against-immediate compare in the window, and sizing a 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as functions inside the function they belong to, so one defect produced a false positive and a shattered function together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no true positive lost on any cell, 135 of 140 cells bit-identical. The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the previous one ended with, so a function reached by falling through inter-function padding was decoded as a continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between functions with, a fall-through cut additionally requires its seed to decode as a function entry on every format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at +3.4% analysis time (reproduced here at +3.84%). The exception directory now also refuses gap candidates it places inside a function. A 64-bit PE carries one `RUNTIME_FUNCTION` record per function, each naming the extent the unwinder needs, so an address strictly inside one is declared to belong to a routine that starts earlier and cannot itself be an entry; only the gap scan ever reaches these addresses, since every one of them sits in a region no reference and no prologue claimed. Two conditions keep it from overreaching. A chained record (`UNW_FLAG_CHAININFO`) describes a fragment of another function, so its first byte is interior too and suppresses on its own, while a primary record's first byte is the entry and stays bookable; and a primary range only suppresses once the analysis has actually recovered the function it names, because a record whose function never analysed is not evidence about what covers the address. The scan then resumes at the extent's end rather than one byte on, which skips the body the record describes instead of re-testing it. Extents are deliberately not merged -- functions are laid out end-to-start, and merging adjacent records would collapse the text section into a handful of spans in which every address but the first reads as interior. Records carved out of an image with no parseable header seed candidates but contribute no extents, which is the distinction that keeps suppression honest: seeding is cheap and self-correcting, suppression is expensive and silent. `USE_PE_X64_PDATA_INTERIOR_GAPS`, on by default. The `is_pe` guard that came with it also stops *seeding* from a non-PE section merely named `.pdata`, which is a candidate-discovery change wider than the title suggests and inert on every bundled image. **Recovery (AArch64):** candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset inside the same corpus. Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary code, which cannot separate these, because a case block is preceded by the previous case's terminating branch and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap scan exists to reach. A follow-up pass then took four candidate-quality defects the BTI series exposed. The interior-BTI test read the candidate snapshot rather than the live function set, so it judged a landing pad against what discovery had proposed instead of against what analysis had recovered. The hoisted-guard heuristic claimed a compare-and-branch could not be a function's first instruction, which is simply false -- on `aarch64_static_xored` the corrected entry sits at `0x40df30`, four bytes ahead of what was booked, and the inter-function `nop` in front of it settles which of the two is the boundary. A metadata-coverage gate now skips the address-materialization scan on an image whose own metadata already named its functions, guarded by a minimum sample count, because a coverage ratio over four resolved call targets says nothing: across the 13 bundled AArch64 images it fires exactly once, on `Turtle_5f9cd91d8d1d` at 98.3% coverage, with output bit-identical either way, so on bundled data it is a work saving with no accuracy consequence. And the ARM64 counterpart of the x64 interior-gap refusal above lands with it: an ARM64 `RUNTIME_FUNCTION` carries no `EndAddress`, so the extent is reconstructed from bits 2-12 of the packed unwind word or from bits 0-17 of the `.xdata` header it points at, both counting instructions. Once an extent is known the rule is identical, which is why the lookup moved into `common/` rather than being written twice; the x64 path keeps the same list, the same bisect and the same running-maximum walk. `USE_PE_ARM64_PDATA_INTERIOR_GAPS`, on by default. Worth knowing that ARM64 PE seeding over-reports on MSVC-built images -- a separated chunk of a routine gets its own record and sets up its own frame, so the entry-shape filter reads it as an entry -- and that no exact test for it is in reach, since 120 of 145 such chunks continue a record with no `.xdata` handler data to name them. **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into(" bool` back through a `bool()` wrapper rather than keeping the annotation deleted; the ruff hook moved to `id: ruff-check`, astral having renamed it and left `ruff` as a legacy alias; `GAP_SEQUENCES` gained a note that it is read in seven places outside the alignment cut, which is what makes a measurement over a change to that table hard to attribute to one caller; and the queue-rebuild test now says it guards against reintroducing scoring into `addTailcallCandidate`, its assertion having become trivially true when that scoring was removed. No escaper output changed, so `ESCAPER_DOWNWARD_COMPATIBILITY` stays at `4.4.5` and `INTEL_PIC_HASH_ESCAPE_VERSION` at `4.3.5`. One serialized value does change shape, in a way worth knowing about downstream: `binweight`'s class default moved from int `0` to float `0.0` while the `ty` errors were being cleared, and `binweight` is written into `toDict()`. Every function with at least one block was already a float, since the per-block accumulation adds `float(...)`, so this is visible only on a function that has no blocks at all -- a zero-function or error report -- where the value now serializes as `0.0` rather than `0`. Nothing reads it as an integer, but a consumer diffing stored reports byte-for-byte will see it. The two items that pass had deliberately left open close here as well, both having needed work upstream of the annotation rather than the annotation itself. `SmdaFunction.blocks` is now typed as what it holds, which restores `getInstructionsForBlock`'s return annotation and forces a decision on the PIC/OPC hash path: it refuses an instruction carrying no bytes rather than defaulting it to a blank the way seven formatting and length-arithmetic sites nearby do, because a blank there would hash a different instruction sequence and the result would still look like a valid hash. And `extract_strings` narrows from `Tuple[str, Any, Any, str]` to `Tuple[str, Optional[int], int, str]`, which required typing `read_string`, `read_go_string` and `derefs` first, and those in turn required declaring `SmdaReport.base_addr`, `.bitness` and `._derefs_cache` -- the reason the item could not be taken as a signature change. Both are behaviour-neutral, and checked as such: PIC/OPC hashes and extracted strings are identical on every fixture that produces them, and the corpus benchmark reports 0 of 155 files differing over 175,946 functions. **Recovery output moves on binaries these paths touch:** eight of the 21 bundled fixtures these paths reach change between v4.5.0 and this release, and the two that carry independent ground truth were scored against it rather than against the contributing PRs' own corpora. `rust_pe_gnu_xored`, against the 2,186 `.text` function symbols its mingw build retains: 2,355 to 2,201 starts, +26 real against -180 false, recall 92.45% to 93.64% and precision 85.82% to 93.00%. The 11-sample ARM64 Mach-O corpus, against `LC_FUNCTION_STARTS` over 2,056 truth functions: 1,787 to 1,818 true positives and 1,113 to 1,107 false ones, recall 86.92% to 88.42%, with no sample losing recall. `aarch64_static_xored` holds at 278 functions with one start corrected four bytes earlier, and `Turtle_5f9cd91d8d1d` drops 12 starts to the inbound-call fix; it carries no `LC_FUNCTION_STARTS`, so those 12 cannot be scored here, but they are part of that change's measured 175 false positives. The remaining 13 are bit-identical. Two headline figures could **not** be reproduced against bundled data and are the contributors' own: the `bti j` 803/0 split, since no bundled fixture contains a single `bti j` word, and the ARM64 PE interior-gap result, since no bundled fixture is an ARM64 PE at all -- of 104 fixture files, 8 are i386 PEs, 4 are AMD64, one is the ReadyToRun image, and none is `0xAA64`. (THX: @r0ny123) * 2026-08-25: v4.5.0 - Function-recovery accuracy pass, MSVC/PDB symbol recovery, synthesis robustness, and the escaper compatibility marker that downstream indexes read. **Recovery (intel/AArch64):** CET `notrack` switch dispatches are now classified as dispatches instead of far jumps and discarded (a single switch previously produced 43 bogus functions); switch tables whose base is register-held are recovered via a bounded `lea` back-walk; jump-table entry scans are bounded against attacker-controlled counts (1049.7 s to 0.002 s) and the address-0 sentinel class is refused at five call sites; the indirect-call backward walk stops resolving against stale registers (branch predecessors, ABI-preserving calls, memory-writing `mov` forms); a function is cut at the CET landing pad it runs into after padding; `push rbp`-opening prologues seed their real entry instead of one byte in; direct calls outrank the inferred alignment floor (recovered `abort` in static glibc). On AArch64, a jump table whose sign-extension is folded into the merging `add` (`add x8, x8, w9, sxtw #2`) is read as signed instead of losing the whole table at its first backward entry, and the constant tracker now drops the registers capstone reports only as implicit writes, so a value held in x30 no longer survives the `bl`/`blr` that clobbers it. On intel, `popa`/`popad` join the instructions known to overwrite eax without naming it, so the syscall-number backtrack stops reporting a `mov eax, N` the pop had already replaced, which is a wrong answer rather than an unresolved one. PE ARM64 exception records are carved from headerless images (0 to 24 functions on veneer-style images) with packed-fragment rejection. Absolute memory operands book data references, which enables a late candidate pass recovering Delphi method-table entries nothing else names (+3 real functions on the corpus, replacing three recoveries that rested on provably wrong stale-register dataflow). Both Delphi method-table walks stop at the first unreadable entry rather than spending a declared 16-bit entry count as an iteration count. The analysis budget is an actual in-pass bound now (polled within passes, latched between them). **Measured against newly bundled labelled ground truth (50 locally built binaries, 11,384 labelled functions): recall 93.781% to 95.098%, body splits 1095 to 306 (-72%), with documented deliberate moves to the `mirai_x64`, `mirai_i386`, `rust_pe_gnu` and `asprox` fixtures.** Median runtime dropped 34% on the 155-file malpedia corpus alongside. **Symbols/demangling:** new vendored MSVC demangler measured per-name against `llvm-undname` on LLVM's own 609-name stress corpus (100% exact spellings, zero third spellings, hostile-input depth and output bounds, shipped with a fuzz target and an attributed reference corpus); `PdbSymbolProvider` rebuilt on `purepdb>=0.3.0`, a declared BSD-compatible pure-Python dependency replacing the optional GPL `pdbparse` import whose failure was silent (three-source record merge: procedures, publics, thunks; OMAP-aware RVA resolution; parent-relative fragment attribution `$+0x`; naming 273/4045 to 3698/4045 functions on sqlite3 x64 and 1/477 to 442/477 on Rust x64); MSVC-decorated PDB public records demangle ahead of the Rust evidence gate; PE symbols demangle Itanium names like their ELF/Mach-O siblings and resolve through `section_idx`; demangled Rust names keep rustc spacing instead of Ghidra-style condensation; punycode identifiers decode non-ASCII text; lief symbol names survive invalid UTF-8 via a shared helper; ELF import names stay binary-encoded for import reconstruction and report merging; Mach-O Swift demangling batches through one toolchain call per image (4.29 s to 0.08 s over the 12-sample corpus, identical output); confirmed-format parse failures across providers and loaders warn instead of vanishing at debug. **Probes/loaders:** bitness decided by REX.W usage statistics over code areas; AArch64 recognized in raw buffers; unrecognized inputs pass through to analysis instead of erroring; opt-in `.eh_frame_hdr`/linker-table candidates on both instruction sets; `WITH_STRINGS` no longer destroys a DEX report's string references; a lief type object's repr can no longer reach a DEX report's type field dressed as a type descriptor, since the ` Date: Tue, 8 Sep 2026 23:12:00 +0200 Subject: [PATCH 6/6] chore: format the v4.5.1 entry as a nested list @r0ny123 asked on #319 whether the changelog could be formatted as a list to make it easier to read. At ~2,700 words on a single line, this entry was the one that provoked the question, so it gets the treatment now rather than waiting for the wider keep-a-changelog adoption tracked in #323. Deliberately narrow: only the v4.5.1 entry changes shape. The other 192 entries keep the one-line format, because converting them is a large, low-value rewrite that loses nuance in translation, and #323 proposes preserving them verbatim under an `Older releases` heading instead. Structure is the release summary as the top-level bullet, one sub-bullet per section, and a further level inside the three sections that cover more than one change -- Recovery (intel) splits into the switch-table fix, the alignment cut and the exception-directory refusal; Recovery (AArch64) into the inbound-call fix, the BTI work and the candidate-quality follow-ups; Housekeeping into the `ty` bump, the `ruff` pin, the five #318 corrections, the `binweight` serialization note and the two #317 typing items. The text is unchanged. Verified mechanically rather than by eye: stripping list markers and collapsing whitespace gives a string identical to the original entry. The two exceptions are disclosed rather than silent -- the first sub-bullet of Recovery (intel) and of Recovery (AArch64) had their opening letter capitalised, because splitting the section header onto its own line left them starting a bullet mid-sentence. Housekeeping's opens on a code span and needed nothing. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr --- CHANGELOG.md | 169 ++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 168 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b16cc8d..1b3a275b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,174 @@ Newest first. Entries carry the release date, the version, and what changed. - * 2026-09-08: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the report's address-space contract written down. **Recovery (intel):** a 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of the table read, falling back to the first register-against-immediate compare in the window, and sizing a 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as functions inside the function they belong to, so one defect produced a false positive and a shattered function together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no true positive lost on any cell, 135 of 140 cells bit-identical. The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the previous one ended with, so a function reached by falling through inter-function padding was decoded as a continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between functions with, a fall-through cut additionally requires its seed to decode as a function entry on every format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at +3.4% analysis time (reproduced here at +3.84%). The exception directory now also refuses gap candidates it places inside a function. A 64-bit PE carries one `RUNTIME_FUNCTION` record per function, each naming the extent the unwinder needs, so an address strictly inside one is declared to belong to a routine that starts earlier and cannot itself be an entry; only the gap scan ever reaches these addresses, since every one of them sits in a region no reference and no prologue claimed. Two conditions keep it from overreaching. A chained record (`UNW_FLAG_CHAININFO`) describes a fragment of another function, so its first byte is interior too and suppresses on its own, while a primary record's first byte is the entry and stays bookable; and a primary range only suppresses once the analysis has actually recovered the function it names, because a record whose function never analysed is not evidence about what covers the address. The scan then resumes at the extent's end rather than one byte on, which skips the body the record describes instead of re-testing it. Extents are deliberately not merged -- functions are laid out end-to-start, and merging adjacent records would collapse the text section into a handful of spans in which every address but the first reads as interior. Records carved out of an image with no parseable header seed candidates but contribute no extents, which is the distinction that keeps suppression honest: seeding is cheap and self-correcting, suppression is expensive and silent. `USE_PE_X64_PDATA_INTERIOR_GAPS`, on by default. The `is_pe` guard that came with it also stops *seeding* from a non-PE section merely named `.pdata`, which is a candidate-discovery change wider than the title suggests and inert on every bundled image. **Recovery (AArch64):** candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset inside the same corpus. Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary code, which cannot separate these, because a case block is preceded by the previous case's terminating branch and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap scan exists to reach. A follow-up pass then took four candidate-quality defects the BTI series exposed. The interior-BTI test read the candidate snapshot rather than the live function set, so it judged a landing pad against what discovery had proposed instead of against what analysis had recovered. The hoisted-guard heuristic claimed a compare-and-branch could not be a function's first instruction, which is simply false -- on `aarch64_static_xored` the corrected entry sits at `0x40df30`, four bytes ahead of what was booked, and the inter-function `nop` in front of it settles which of the two is the boundary. A metadata-coverage gate now skips the address-materialization scan on an image whose own metadata already named its functions, guarded by a minimum sample count, because a coverage ratio over four resolved call targets says nothing: across the 13 bundled AArch64 images it fires exactly once, on `Turtle_5f9cd91d8d1d` at 98.3% coverage, with output bit-identical either way, so on bundled data it is a work saving with no accuracy consequence. And the ARM64 counterpart of the x64 interior-gap refusal above lands with it: an ARM64 `RUNTIME_FUNCTION` carries no `EndAddress`, so the extent is reconstructed from bits 2-12 of the packed unwind word or from bits 0-17 of the `.xdata` header it points at, both counting instructions. Once an extent is known the rule is identical, which is why the lookup moved into `common/` rather than being written twice; the x64 path keeps the same list, the same bisect and the same running-maximum walk. `USE_PE_ARM64_PDATA_INTERIOR_GAPS`, on by default. Worth knowing that ARM64 PE seeding over-reports on MSVC-built images -- a separated chunk of a routine gets its own record and sets up its own frame, so the entry-shape filter reads it as an entry -- and that no exact test for it is in reach, since 120 of 145 such chunks continue a record with no `.xdata` handler data to name them. **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into(" bool` back through a `bool()` wrapper rather than keeping the annotation deleted; the ruff hook moved to `id: ruff-check`, astral having renamed it and left `ruff` as a legacy alias; `GAP_SEQUENCES` gained a note that it is read in seven places outside the alignment cut, which is what makes a measurement over a change to that table hard to attribute to one caller; and the queue-rebuild test now says it guards against reintroducing scoring into `addTailcallCandidate`, its assertion having become trivially true when that scoring was removed. No escaper output changed, so `ESCAPER_DOWNWARD_COMPATIBILITY` stays at `4.4.5` and `INTEL_PIC_HASH_ESCAPE_VERSION` at `4.3.5`. One serialized value does change shape, in a way worth knowing about downstream: `binweight`'s class default moved from int `0` to float `0.0` while the `ty` errors were being cleared, and `binweight` is written into `toDict()`. Every function with at least one block was already a float, since the per-block accumulation adds `float(...)`, so this is visible only on a function that has no blocks at all -- a zero-function or error report -- where the value now serializes as `0.0` rather than `0`. Nothing reads it as an integer, but a consumer diffing stored reports byte-for-byte will see it. The two items that pass had deliberately left open close here as well, both having needed work upstream of the annotation rather than the annotation itself. `SmdaFunction.blocks` is now typed as what it holds, which restores `getInstructionsForBlock`'s return annotation and forces a decision on the PIC/OPC hash path: it refuses an instruction carrying no bytes rather than defaulting it to a blank the way seven formatting and length-arithmetic sites nearby do, because a blank there would hash a different instruction sequence and the result would still look like a valid hash. And `extract_strings` narrows from `Tuple[str, Any, Any, str]` to `Tuple[str, Optional[int], int, str]`, which required typing `read_string`, `read_go_string` and `derefs` first, and those in turn required declaring `SmdaReport.base_addr`, `.bitness` and `._derefs_cache` -- the reason the item could not be taken as a signature change. Both are behaviour-neutral, and checked as such: PIC/OPC hashes and extracted strings are identical on every fixture that produces them, and the corpus benchmark reports 0 of 155 files differing over 175,946 functions. **Recovery output moves on binaries these paths touch:** eight of the 21 bundled fixtures these paths reach change between v4.5.0 and this release, and the two that carry independent ground truth were scored against it rather than against the contributing PRs' own corpora. `rust_pe_gnu_xored`, against the 2,186 `.text` function symbols its mingw build retains: 2,355 to 2,201 starts, +26 real against -180 false, recall 92.45% to 93.64% and precision 85.82% to 93.00%. The 11-sample ARM64 Mach-O corpus, against `LC_FUNCTION_STARTS` over 2,056 truth functions: 1,787 to 1,818 true positives and 1,113 to 1,107 false ones, recall 86.92% to 88.42%, with no sample losing recall. `aarch64_static_xored` holds at 278 functions with one start corrected four bytes earlier, and `Turtle_5f9cd91d8d1d` drops 12 starts to the inbound-call fix; it carries no `LC_FUNCTION_STARTS`, so those 12 cannot be scored here, but they are part of that change's measured 175 false positives. The remaining 13 are bit-identical. Two headline figures could **not** be reproduced against bundled data and are the contributors' own: the `bti j` 803/0 split, since no bundled fixture contains a single `bti j` word, and the ARM64 PE interior-gap result, since no bundled fixture is an ARM64 PE at all -- of 104 fixture files, 8 are i386 PEs, 4 are AMD64, one is the ReadyToRun image, and none is `0xAA64`. (THX: @r0ny123) + * 2026-09-08: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the + report's address-space contract written down. + * **Recovery (intel):** + * A 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and + `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of + the table read, falling back to the first register-against-immediate compare in the window, and sizing a + 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as + functions inside the function they belong to, so one defect produced a false positive and a shattered function + together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a + register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction + the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not + a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no + true positive lost on any cell, 135 of 140 cells bit-identical. + * The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- + on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the + previous one ended with, so a function reached by falling through inter-function padding was decoded as a + continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as + a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and + the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between + functions with, a fall-through cut additionally requires its seed to decode as a function entry on every + format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 + 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not + only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% + and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at + +3.4% analysis time (reproduced here at +3.84%). + * The exception directory now also refuses gap candidates it places inside a function. A 64-bit PE carries one + `RUNTIME_FUNCTION` record per function, each naming the extent the unwinder needs, so an address strictly + inside one is declared to belong to a routine that starts earlier and cannot itself be an entry; only the gap + scan ever reaches these addresses, since every one of them sits in a region no reference and no prologue + claimed. Two conditions keep it from overreaching. A chained record (`UNW_FLAG_CHAININFO`) describes a fragment + of another function, so its first byte is interior too and suppresses on its own, while a primary record's + first byte is the entry and stays bookable; and a primary range only suppresses once the analysis has actually + recovered the function it names, because a record whose function never analysed is not evidence about what + covers the address. The scan then resumes at the extent's end rather than one byte on, which skips the body the + record describes instead of re-testing it. Extents are deliberately not merged -- functions are laid out + end-to-start, and merging adjacent records would collapse the text section into a handful of spans in which + every address but the first reads as interior. Records carved out of an image with no parseable header seed + candidates but contribute no extents, which is the distinction that keeps suppression honest: seeding is cheap + and self-correcting, suppression is expensive and silent. `USE_PE_X64_PDATA_INTERIOR_GAPS`, on by default. The + `is_pe` guard that came with it also stops *seeding* from a non-PE section merely named `.pdata`, which is a + candidate-discovery change wider than the title suggests and inert on every bundled image. + * **Recovery (AArch64):** + * Candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the + AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound + call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that + an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses + went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the + smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is + not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the + alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured + across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on + the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function + and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset + inside the same corpus. + * Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite + things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only + pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target + - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary + code, which cannot separate these, because a case block is preceded by the previous case's terminating branch + and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads + the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact + rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true + and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false + positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that + fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real + entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the + false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at + `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps + and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap + scan exists to reach. + * A follow-up pass then took four candidate-quality defects the BTI series exposed. The interior-BTI test read + the candidate snapshot rather than the live function set, so it judged a landing pad against what discovery had + proposed instead of against what analysis had recovered. The hoisted-guard heuristic claimed a + compare-and-branch could not be a function's first instruction, which is simply false -- on + `aarch64_static_xored` the corrected entry sits at `0x40df30`, four bytes ahead of what was booked, and the + inter-function `nop` in front of it settles which of the two is the boundary. A metadata-coverage gate now + skips the address-materialization scan on an image whose own metadata already named its functions, guarded by a + minimum sample count, because a coverage ratio over four resolved call targets says nothing: across the 13 + bundled AArch64 images it fires exactly once, on `Turtle_5f9cd91d8d1d` at 98.3% coverage, with output + bit-identical either way, so on bundled data it is a work saving with no accuracy consequence. And the ARM64 + counterpart of the x64 interior-gap refusal above lands with it: an ARM64 `RUNTIME_FUNCTION` carries no + `EndAddress`, so the extent is reconstructed from bits 2-12 of the packed unwind word or from bits 0-17 of the + `.xdata` header it points at, both counting instructions. Once an extent is known the rule is identical, which + is why the lookup moved into `common/` rather than being written twice; the x64 path keeps the same list, the + same bisect and the same running-maximum walk. `USE_PE_ARM64_PDATA_INTERIOR_GAPS`, on by default. Worth knowing + that ARM64 PE seeding over-reports on MSVC-built images -- a separated chunk of a routine gets its own record + and sets up its own frame, so the entry-shape filter reads it as an entry -- and that no exact test for it is + in reach, since 120 of 145 such chunks continue a record with no `.xdata` handler data to name them. + * **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps + how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but + `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose + entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into(" bool` back through a `bool()` wrapper rather than keeping the + annotation deleted; the ruff hook moved to `id: ruff-check`, astral having renamed it and left `ruff` as a + legacy alias; `GAP_SEQUENCES` gained a note that it is read in seven places outside the alignment cut, which is + what makes a measurement over a change to that table hard to attribute to one caller; and the queue-rebuild + test now says it guards against reintroducing scoring into `addTailcallCandidate`, its assertion having become + trivially true when that scoring was removed. No escaper output changed, so `ESCAPER_DOWNWARD_COMPATIBILITY` + stays at `4.4.5` and `INTEL_PIC_HASH_ESCAPE_VERSION` at `4.3.5`. + * One serialized value does change shape, in a way worth knowing about downstream: `binweight`'s class default + moved from int `0` to float `0.0` while the `ty` errors were being cleared, and `binweight` is written into + `toDict()`. Every function with at least one block was already a float, since the per-block accumulation adds + `float(...)`, so this is visible only on a function that has no blocks at all -- a zero-function or error + report -- where the value now serializes as `0.0` rather than `0`. Nothing reads it as an integer, but a + consumer diffing stored reports byte-for-byte will see it. + * The two items that pass had deliberately left open close here as well, both having needed work upstream of the + annotation rather than the annotation itself. `SmdaFunction.blocks` is now typed as what it holds, which + restores `getInstructionsForBlock`'s return annotation and forces a decision on the PIC/OPC hash path: it + refuses an instruction carrying no bytes rather than defaulting it to a blank the way seven formatting and + length-arithmetic sites nearby do, because a blank there would hash a different instruction sequence and the + result would still look like a valid hash. And `extract_strings` narrows from `Tuple[str, Any, Any, str]` to + `Tuple[str, Optional[int], int, str]`, which required typing `read_string`, `read_go_string` and `derefs` + first, and those in turn required declaring `SmdaReport.base_addr`, `.bitness` and `._derefs_cache` -- the + reason the item could not be taken as a signature change. Both are behaviour-neutral, and checked as such: + PIC/OPC hashes and extracted strings are identical on every fixture that produces them, and the corpus + benchmark reports 0 of 155 files differing over 175,946 functions. + * **Recovery output moves on binaries these paths touch:** eight of the 21 bundled fixtures these paths reach + change between v4.5.0 and this release, and the two that carry independent ground truth were scored against it + rather than against the contributing PRs' own corpora. `rust_pe_gnu_xored`, against the 2,186 `.text` function + symbols its mingw build retains: 2,355 to 2,201 starts, +26 real against -180 false, recall 92.45% to 93.64% and + precision 85.82% to 93.00%. The 11-sample ARM64 Mach-O corpus, against `LC_FUNCTION_STARTS` over 2,056 truth + functions: 1,787 to 1,818 true positives and 1,113 to 1,107 false ones, recall 86.92% to 88.42%, with no sample + losing recall. `aarch64_static_xored` holds at 278 functions with one start corrected four bytes earlier, and + `Turtle_5f9cd91d8d1d` drops 12 starts to the inbound-call fix; it carries no `LC_FUNCTION_STARTS`, so those 12 + cannot be scored here, but they are part of that change's measured 175 false positives. The remaining 13 are + bit-identical. Two headline figures could **not** be reproduced against bundled data and are the contributors' + own: the `bti j` 803/0 split, since no bundled fixture contains a single `bti j` word, and the ARM64 PE + interior-gap result, since no bundled fixture is an ARM64 PE at all -- of 104 fixture files, 8 are i386 PEs, 4 + are AMD64, one is the ReadyToRun image, and none is `0xAA64`. (THX: @r0ny123) * 2026-08-25: v4.5.0 - Function-recovery accuracy pass, MSVC/PDB symbol recovery, synthesis robustness, and the escaper compatibility marker that downstream indexes read. **Recovery (intel/AArch64):** CET `notrack` switch dispatches are now classified as dispatches instead of far jumps and discarded (a single switch previously produced 43 bogus functions); switch tables whose base is register-held are recovered via a bounded `lea` back-walk; jump-table entry scans are bounded against attacker-controlled counts (1049.7 s to 0.002 s) and the address-0 sentinel class is refused at five call sites; the indirect-call backward walk stops resolving against stale registers (branch predecessors, ABI-preserving calls, memory-writing `mov` forms); a function is cut at the CET landing pad it runs into after padding; `push rbp`-opening prologues seed their real entry instead of one byte in; direct calls outrank the inferred alignment floor (recovered `abort` in static glibc). On AArch64, a jump table whose sign-extension is folded into the merging `add` (`add x8, x8, w9, sxtw #2`) is read as signed instead of losing the whole table at its first backward entry, and the constant tracker now drops the registers capstone reports only as implicit writes, so a value held in x30 no longer survives the `bl`/`blr` that clobbers it. On intel, `popa`/`popad` join the instructions known to overwrite eax without naming it, so the syscall-number backtrack stops reporting a `mov eax, N` the pop had already replaced, which is a wrong answer rather than an unresolved one. PE ARM64 exception records are carved from headerless images (0 to 24 functions on veneer-style images) with packed-fragment rejection. Absolute memory operands book data references, which enables a late candidate pass recovering Delphi method-table entries nothing else names (+3 real functions on the corpus, replacing three recoveries that rested on provably wrong stale-register dataflow). Both Delphi method-table walks stop at the first unreadable entry rather than spending a declared 16-bit entry count as an iteration count. The analysis budget is an actual in-pass bound now (polled within passes, latched between them). **Measured against newly bundled labelled ground truth (50 locally built binaries, 11,384 labelled functions): recall 93.781% to 95.098%, body splits 1095 to 306 (-72%), with documented deliberate moves to the `mirai_x64`, `mirai_i386`, `rust_pe_gnu` and `asprox` fixtures.** Median runtime dropped 34% on the 155-file malpedia corpus alongside. **Symbols/demangling:** new vendored MSVC demangler measured per-name against `llvm-undname` on LLVM's own 609-name stress corpus (100% exact spellings, zero third spellings, hostile-input depth and output bounds, shipped with a fuzz target and an attributed reference corpus); `PdbSymbolProvider` rebuilt on `purepdb>=0.3.0`, a declared BSD-compatible pure-Python dependency replacing the optional GPL `pdbparse` import whose failure was silent (three-source record merge: procedures, publics, thunks; OMAP-aware RVA resolution; parent-relative fragment attribution `$+0x`; naming 273/4045 to 3698/4045 functions on sqlite3 x64 and 1/477 to 442/477 on Rust x64); MSVC-decorated PDB public records demangle ahead of the Rust evidence gate; PE symbols demangle Itanium names like their ELF/Mach-O siblings and resolve through `section_idx`; demangled Rust names keep rustc spacing instead of Ghidra-style condensation; punycode identifiers decode non-ASCII text; lief symbol names survive invalid UTF-8 via a shared helper; ELF import names stay binary-encoded for import reconstruction and report merging; Mach-O Swift demangling batches through one toolchain call per image (4.29 s to 0.08 s over the 12-sample corpus, identical output); confirmed-format parse failures across providers and loaders warn instead of vanishing at debug. **Probes/loaders:** bitness decided by REX.W usage statistics over code areas; AArch64 recognized in raw buffers; unrecognized inputs pass through to analysis instead of erroring; opt-in `.eh_frame_hdr`/linker-table candidates on both instruction sets; `WITH_STRINGS` no longer destroys a DEX report's string references; a lief type object's repr can no longer reach a DEX report's type field dressed as a type descriptor, since the `