diff --git a/CHANGELOG.md b/CHANGELOG.md index ec218d57..1b3a275b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,175 @@ Newest first. Entries carry the release date, the version, and what changed. + * 2026-09-08: v4.5.1 - Function-recovery fixes across intel and AArch64, a fuzz-found synthesis crash, and the + report's address-space contract written down. + * **Recovery (intel):** + * A 64-bit relative switch dispatch ends by adding the table's base to the entry the table read produced, and + `_findJumpTableSize` treated that `add` as a redefinition -- dropping the index tie one instruction short of + the table read, falling back to the first register-against-immediate compare in the window, and sizing a + 27-case table at 2. The 25 unreferenced case bodies then reached the gap scan, which booked five of them as + functions inside the function they belong to, so one defect produced a false positive and a shattered function + together. The tie is now carried across the base add in both spellings a compiler writes (`add rax, rdi` with a + register or memory source, and a `lea` whose address reads the register it writes), and only on the instruction + the branch actually reads: an `add` further back is index arithmetic, and a compare bounding one summand is not + a bound on the sum. Measured on 140 built C/C++ cells (117,654 truth functions): 20 false positives removed, no + true positive lost on any cell, 135 of 140 cells bit-identical. + * The alignment cut also gained its second way in. It had exactly one before -- padding that follows a `call` -- + on the theory that a noreturn call is what leaves padding behind; GCC pads between functions whatever the + previous one ended with, so a function reached by falling through inter-function padding was decoded as a + continuation of its predecessor and the two came back as one. Padding is now evidence in its own right, read as + a whole run: every byte from the current address to the next 16-byte boundary has to be alignment filler and + the run has to stop at that boundary. Because GCC aligns loop heads with the same encodings it pads between + functions with, a fall-through cut additionally requires its seed to decode as a function entry on every + format, where the call path keeps the old exemption. Two CS-prefixed GCC/binutils multi-byte nops (`2e 8d 74 26 + 00` and `2e 8d b4 26 00 00 00 00`) join `GAP_SEQUENCES`, which sharpens every padding test that reads it, not + only the new one. Measured on 296,225 ground-truth functions over 191 samples: pooled recall 97.29% to 98.31% + and precision 92.83% to 93.70%, +3,009 true starts against -2,676 false ones, all 47 Go cells bit-identical, at + +3.4% analysis time (reproduced here at +3.84%). + * The exception directory now also refuses gap candidates it places inside a function. A 64-bit PE carries one + `RUNTIME_FUNCTION` record per function, each naming the extent the unwinder needs, so an address strictly + inside one is declared to belong to a routine that starts earlier and cannot itself be an entry; only the gap + scan ever reaches these addresses, since every one of them sits in a region no reference and no prologue + claimed. Two conditions keep it from overreaching. A chained record (`UNW_FLAG_CHAININFO`) describes a fragment + of another function, so its first byte is interior too and suppresses on its own, while a primary record's + first byte is the entry and stays bookable; and a primary range only suppresses once the analysis has actually + recovered the function it names, because a record whose function never analysed is not evidence about what + covers the address. The scan then resumes at the extent's end rather than one byte on, which skips the body the + record describes instead of re-testing it. Extents are deliberately not merged -- functions are laid out + end-to-start, and merging adjacent records would collapse the text section into a handful of spans in which + every address but the first reads as interior. Records carved out of an image with no parseable header seed + candidates but contribute no extents, which is the distinction that keeps suppression honest: seeding is cheap + and self-correcting, suppression is expensive and silent. `USE_PE_X64_PDATA_INTERIOR_GAPS`, on by default. The + `is_pe` guard that came with it also stops *seeding* from a non-PE section merely named `.pdata`, which is a + candidate-discovery change wider than the title suggests and inert on every bundled image. + * **Recovery (AArch64):** + * Candidate discovery seeds a function start at the target of a backward branch or a short no-frame stub, and the + AArch64 candidate manager recorded the branching instruction as an inbound *call* reference for it. An inbound + call is one of exactly two things exempt from the inferred alignment floor -- the image's own declaration that + an address is an entry -- so the seed manufactured the evidence the floor trusts, and mid-function addresses + went through it. That is worse than a stray false positive: on a routine only the gap scan recovers, the + smuggled interior address is analysed first, takes the bytes, and the real start is never reported. A branch is + not a call, so no reference is recorded now; the address is still discovered, queued and analysed, on the + alignment and prologue it actually has. The shared implementation in `common/` never recorded one. Measured + across three corpora: 175 false positives removed and 3 functions recovered, with no corpus losing recall; on + the bundled 11-sample ARM64 Mach-O corpus the totals are flat because `LockBit_3e4bbd21756a` gains a function + and `RustyPages_e98756472404` loses `0x10000548c`, which is the single recall cost anywhere and is offset + inside the same corpus. + * Separately, candidate discovery read all four BTI forms as entry prologues, and two of them mean opposite + things: `bti j` permits a target reached by `br` and never one reached by `blr`, so a call landing on a J-only + pad faults and the compiler that wrote J was naming an interior label - a switch case or a computed-goto target + - rather than an entry. The existing guard reads the word before a pad and refuses one that follows ordinary + code, which cannot separate these, because a case block is preceded by the previous case's terminating branch + and that is exactly the boundary shape the guard accepts. `USE_AARCH64_BTI_TARGET_TYPE`, on by default, reads + the target type instead: over 1,308 prologue-sole bookings on 72 built AArch64 ELF cells the split is exact + rather than statistical, `bti j` accounting for 803 false positives and no true ones against `bti c`'s 505 true + and none false. Precision moves 87.596 to 89.186 with recall unchanged to three decimals and 2,941 false + positives removed, which is more than the 803 bookings because a wrong entry also starts an analysis that + fragments the routine it sits inside. `bti jc` is deliberately untouched: it permits `blr`, so it can be a real + entry. Refusing a pad then has to resume past the block the pad labels rather than one instruction on, or the + false positive simply moves four bytes onto the pad's own first body instruction - and that walk stops at + `brk`/`hlt`/`udf` as well as at `ret`/`br`/`b`, because `analyzeInstruction` ends a function on all three traps + and a walk that reads past one skips whatever lies behind it, which is exactly the unreferenced routine the gap + scan exists to reach. + * A follow-up pass then took four candidate-quality defects the BTI series exposed. The interior-BTI test read + the candidate snapshot rather than the live function set, so it judged a landing pad against what discovery had + proposed instead of against what analysis had recovered. The hoisted-guard heuristic claimed a + compare-and-branch could not be a function's first instruction, which is simply false -- on + `aarch64_static_xored` the corrected entry sits at `0x40df30`, four bytes ahead of what was booked, and the + inter-function `nop` in front of it settles which of the two is the boundary. A metadata-coverage gate now + skips the address-materialization scan on an image whose own metadata already named its functions, guarded by a + minimum sample count, because a coverage ratio over four resolved call targets says nothing: across the 13 + bundled AArch64 images it fires exactly once, on `Turtle_5f9cd91d8d1d` at 98.3% coverage, with output + bit-identical either way, so on bundled data it is a work saving with no accuracy consequence. And the ARM64 + counterpart of the x64 interior-gap refusal above lands with it: an ARM64 `RUNTIME_FUNCTION` carries no + `EndAddress`, so the extent is reconstructed from bits 2-12 of the packed unwind word or from bits 0-17 of the + `.xdata` header it points at, both counting instructions. Once an extent is known the rule is identical, which + is why the lookup moved into `common/` rather than being written twice; the x64 path keeps the same list, the + same bisect and the same running-maximum walk. `USE_PE_ARM64_PDATA_INTERIOR_GAPS`, on by default. Worth knowing + that ARM64 PE seeding over-reports on MSVC-built images -- a separated chunk of a routine gets its own record + and sets up its own frame, so the entry-shape filter reads it as an entry -- and that no exact test for it is + in reach, since 120 of 145 such chunks continue a record with no `.xdata` handler data to name them. + * **Synthesis:** a report's entry point is the one image field no span check bounds. `_resolveFunctionOffsets` caps + how far apart the *functions* may sit, which is what keeps every RVA derived from them inside a header field, but + `oep` is not a function offset and reached the header packers unchecked. `fuzz_synthesis` found it: a PE whose + entry sits more than 4 GiB above the image base raised `struct.error` out of `struct.pack_into(" bool` back through a `bool()` wrapper rather than keeping the + annotation deleted; the ruff hook moved to `id: ruff-check`, astral having renamed it and left `ruff` as a + legacy alias; `GAP_SEQUENCES` gained a note that it is read in seven places outside the alignment cut, which is + what makes a measurement over a change to that table hard to attribute to one caller; and the queue-rebuild + test now says it guards against reintroducing scoring into `addTailcallCandidate`, its assertion having become + trivially true when that scoring was removed. No escaper output changed, so `ESCAPER_DOWNWARD_COMPATIBILITY` + stays at `4.4.5` and `INTEL_PIC_HASH_ESCAPE_VERSION` at `4.3.5`. + * One serialized value does change shape, in a way worth knowing about downstream: `binweight`'s class default + moved from int `0` to float `0.0` while the `ty` errors were being cleared, and `binweight` is written into + `toDict()`. Every function with at least one block was already a float, since the per-block accumulation adds + `float(...)`, so this is visible only on a function that has no blocks at all -- a zero-function or error + report -- where the value now serializes as `0.0` rather than `0`. Nothing reads it as an integer, but a + consumer diffing stored reports byte-for-byte will see it. + * The two items that pass had deliberately left open close here as well, both having needed work upstream of the + annotation rather than the annotation itself. `SmdaFunction.blocks` is now typed as what it holds, which + restores `getInstructionsForBlock`'s return annotation and forces a decision on the PIC/OPC hash path: it + refuses an instruction carrying no bytes rather than defaulting it to a blank the way seven formatting and + length-arithmetic sites nearby do, because a blank there would hash a different instruction sequence and the + result would still look like a valid hash. And `extract_strings` narrows from `Tuple[str, Any, Any, str]` to + `Tuple[str, Optional[int], int, str]`, which required typing `read_string`, `read_go_string` and `derefs` + first, and those in turn required declaring `SmdaReport.base_addr`, `.bitness` and `._derefs_cache` -- the + reason the item could not be taken as a signature change. Both are behaviour-neutral, and checked as such: + PIC/OPC hashes and extracted strings are identical on every fixture that produces them, and the corpus + benchmark reports 0 of 155 files differing over 175,946 functions. + * **Recovery output moves on binaries these paths touch:** eight of the 21 bundled fixtures these paths reach + change between v4.5.0 and this release, and the two that carry independent ground truth were scored against it + rather than against the contributing PRs' own corpora. `rust_pe_gnu_xored`, against the 2,186 `.text` function + symbols its mingw build retains: 2,355 to 2,201 starts, +26 real against -180 false, recall 92.45% to 93.64% and + precision 85.82% to 93.00%. The 11-sample ARM64 Mach-O corpus, against `LC_FUNCTION_STARTS` over 2,056 truth + functions: 1,787 to 1,818 true positives and 1,113 to 1,107 false ones, recall 86.92% to 88.42%, with no sample + losing recall. `aarch64_static_xored` holds at 278 functions with one start corrected four bytes earlier, and + `Turtle_5f9cd91d8d1d` drops 12 starts to the inbound-call fix; it carries no `LC_FUNCTION_STARTS`, so those 12 + cannot be scored here, but they are part of that change's measured 175 false positives. The remaining 13 are + bit-identical. Two headline figures could **not** be reproduced against bundled data and are the contributors' + own: the `bti j` 803/0 split, since no bundled fixture contains a single `bti j` word, and the ARM64 PE + interior-gap result, since no bundled fixture is an ARM64 PE at all -- of 104 fixture files, 8 are i386 PEs, 4 + are AMD64, one is the ReadyToRun image, and none is `0xAA64`. (THX: @r0ny123) + * 2026-08-25: v4.5.0 - Function-recovery accuracy pass, MSVC/PDB symbol recovery, synthesis robustness, and the escaper compatibility marker that downstream indexes read. **Recovery (intel/AArch64):** CET `notrack` switch dispatches are now classified as dispatches instead of far jumps and discarded (a single switch previously produced 43 bogus functions); switch tables whose base is register-held are recovered via a bounded `lea` back-walk; jump-table entry scans are bounded against attacker-controlled counts (1049.7 s to 0.002 s) and the address-0 sentinel class is refused at five call sites; the indirect-call backward walk stops resolving against stale registers (branch predecessors, ABI-preserving calls, memory-writing `mov` forms); a function is cut at the CET landing pad it runs into after padding; `push rbp`-opening prologues seed their real entry instead of one byte in; direct calls outrank the inferred alignment floor (recovered `abort` in static glibc). On AArch64, a jump table whose sign-extension is folded into the merging `add` (`add x8, x8, w9, sxtw #2`) is read as signed instead of losing the whole table at its first backward entry, and the constant tracker now drops the registers capstone reports only as implicit writes, so a value held in x30 no longer survives the `bl`/`blr` that clobbers it. On intel, `popa`/`popad` join the instructions known to overwrite eax without naming it, so the syscall-number backtrack stops reporting a `mov eax, N` the pop had already replaced, which is a wrong answer rather than an unresolved one. PE ARM64 exception records are carved from headerless images (0 to 24 functions on veneer-style images) with packed-fragment rejection. Absolute memory operands book data references, which enables a late candidate pass recovering Delphi method-table entries nothing else names (+3 real functions on the corpus, replacing three recoveries that rested on provably wrong stale-register dataflow). Both Delphi method-table walks stop at the first unreadable entry rather than spending a declared 16-bit entry count as an iteration count. The analysis budget is an actual in-pass bound now (polled within passes, latched between them). **Measured against newly bundled labelled ground truth (50 locally built binaries, 11,384 labelled functions): recall 93.781% to 95.098%, body splits 1095 to 306 (-72%), with documented deliberate moves to the `mirai_x64`, `mirai_i386`, `rust_pe_gnu` and `asprox` fixtures.** Median runtime dropped 34% on the 155-file malpedia corpus alongside. **Symbols/demangling:** new vendored MSVC demangler measured per-name against `llvm-undname` on LLVM's own 609-name stress corpus (100% exact spellings, zero third spellings, hostile-input depth and output bounds, shipped with a fuzz target and an attributed reference corpus); `PdbSymbolProvider` rebuilt on `purepdb>=0.3.0`, a declared BSD-compatible pure-Python dependency replacing the optional GPL `pdbparse` import whose failure was silent (three-source record merge: procedures, publics, thunks; OMAP-aware RVA resolution; parent-relative fragment attribution `$+0x`; naming 273/4045 to 3698/4045 functions on sqlite3 x64 and 1/477 to 442/477 on Rust x64); MSVC-decorated PDB public records demangle ahead of the Rust evidence gate; PE symbols demangle Itanium names like their ELF/Mach-O siblings and resolve through `section_idx`; demangled Rust names keep rustc spacing instead of Ghidra-style condensation; punycode identifiers decode non-ASCII text; lief symbol names survive invalid UTF-8 via a shared helper; ELF import names stay binary-encoded for import reconstruction and report merging; Mach-O Swift demangling batches through one toolchain call per image (4.29 s to 0.08 s over the 12-sample corpus, identical output); confirmed-format parse failures across providers and loaders warn instead of vanishing at debug. **Probes/loaders:** bitness decided by REX.W usage statistics over code areas; AArch64 recognized in raw buffers; unrecognized inputs pass through to analysis instead of erroring; opt-in `.eh_frame_hdr`/linker-table candidates on both instruction sets; `WITH_STRINGS` no longer destroys a DEX report's string references; a lief type object's repr can no longer reach a DEX report's type field dressed as a type descriptor, since the `