perf(intel): cut on padding that falls through, not only on padding after a call - #304
Merged
danielplohmann merged 2 commits intoSep 7, 2026
Conversation
r0ny123
force-pushed
the
upstream-intel-alignment-padding-cut
branch
from
August 30, 2026 03:36
d09cb39 to
9f76d13
Compare
…fter a call
A function reached by falling through GCC's inter-function padding was decoded as
a continuation of its predecessor, and the two were reported as one.
The alignment cut had exactly one way in: the previous instruction had to be a
call, on the theory that padding follows a function that ended in a noreturn
call. GCC pads between functions whatever the previous one ended with, so
everything reached by falling through such padding was invisible to the cut.
The other way in is the padding itself. _paddingFillsToAlignment reads the run
whole: every byte from the current address to the next 16-byte boundary has to
be alignment filler, and the boundary itself has to be where the run stops.
Reading the run rather than its first encoding is what carries the change --
1,598 of the 1,778 addresses it refuses that a bare "not on a boundary" test
would have cut are not functions, and on Go it is 1,364 of 1,365.
That is still not enough on its own. GCC aligns loop heads with the same
encodings it pads between functions with, and both runs end on the boundary, so
padding reached by falling through now needs its seed to decode as a function
entry on every format. Only the call path keeps the old exemption, because the
call is the prior that the function ended. Without this the ELF fixtures gained
five loop heads inside one unrolled string routine.
Measured over 296,225 ground-truth functions in 191 samples (gcc/clang/mingw
C/C++, Go, Rust; 8 optimisation and link configurations each):
TPR 97.29% -> 98.31% (+3,009 true starts, 2 lost)
PPV 92.83% -> 93.70% (-2,676 false starts)
The C/C++ and Rust corpora carry all of it; Go is unchanged, all 47 cells
bit-identical, which is what an x86 padding rule should do to images whose
pclntab already names every function. On the 158-sample malware corpus, two
files change by one address each. Analysis time is +3.4% on a paired run of ten
of those samples.
The table also gains the two CS-prefixed nops binutils emits for widths 5 and 8,
which it had for the 0x90-prefixed forms but not these.
r0ny123
force-pushed
the
upstream-intel-alignment-padding-cut
branch
from
August 30, 2026 03:45
9f76d13 to
2643f9b
Compare
Keep the 0.0.74 bump and satisfy the new unsound-return/assignment/yield rules with annotations on the public DTO and helper surfaces. Ignore ty in Dependabot so the next patch release cannot redden master unattended.
This was referenced Sep 7, 2026
Merged
This was referenced Sep 8, 2026
danielplohmann
added a commit
that referenced
this pull request
Sep 8, 2026
* chore: draft the v4.5.1 release notes The changelog entry for what has landed since v4.5.0 (6cc5aca): #302, #303, #304, #305, #306, #307, #308, #310 and #316, plus the Dependabot ty bump 572acd7 that #302 had to answer for. Held as a draft rather than a release. The version is deliberately NOT bumped here -- #309, #311 and #312 are reviewed and wanted but waiting on a rebase, and splitting one coherent accuracy pass across two releases makes both entries weaker. What this commit does is stop the writeup living in a scratchpad while that happens. To finish it: add sections for #309, #311 and #312, bump VERSION in src/smda/SmdaConfig.py and __version__ in src/smda/__init__.py, and move the date to the actual release day. No escaper output changed anywhere in this set -- the only edit to intel/definitions.py adds two GAP_SEQUENCES entries, which feed padding detection and not escaping, and testEscaperFingerprint passes unchanged -- so ESCAPER_DOWNWARD_COMPATIBILITY stays at 4.4.5 and INTEL_PIC_HASH_ESCAPE_VERSION at 4.3.5, and no report needs reprocessing. Figures are the contributors' own except where this session reproduced them, and the two that were reproduced are stated as measured here: the padding cut's analysis-time cost (+3.4% claimed, +3.84% measured) and the fixture movement on rust_pe_gnu_xored (+26 real starts against +8 false, scored against that fixture's own COFF symbol table rather than the corpus macro mean). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: fold #318 into the v4.5.1 release notes #318 landed as 395c88d after this entry was written. Its five corrections go under Housekeeping: the README's metadata.language claim, the restored `-> bool`, the ruff-check hook rename, the GAP_SEQUENCES reach note, and the docstring on the queue-rebuild test. Keeping this current as things merge is the point of the branch existing, so it does not go stale between now and the release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: cut v4.5.1 Finishes the draft now that #309, #311 and #312 have landed. Adds their sections, bumps `VERSION` and `__version__` to 4.5.1, and moves the date to the release day. The three new entries: #309's exception-directory refusal of interior gap candidates on x64, with the chained-versus-primary distinction and the seeding-is-not-suppression principle that keeps carved records out of it; #311's four candidate-quality defects, including the corrected entry at `0x40df30` on `aarch64_static_xored` and the metadata-coverage gate; and #312's ARM64 counterpart, which reconstructs the extent an ARM64 `RUNTIME_FUNCTION` does not carry and hoists the shared lookup into `common/`. `USE_MACHO_ADDRESS_REF_CANDIDATES` gets its own **Defaults changed** section. It is the only default whose value moves in this release, it changes recovery output on Mach-O input with no action by the caller, and a default going from off to on inside a follow-ups PR is exactly the kind of thing that gets lost. The closing movement section is rewritten to describe v4.5.0 -> v4.5.1 rather than any single PR, and both figures in it were measured on this tree against independent truth rather than quoted: rust_pe_gnu_xored, vs its 2,186 retained COFF .text function symbols 2,355 -> 2,201 starts, +26 real / -180 false recall 92.452% -> 93.641% precision 85.817% -> 93.003% 11-sample ARM64 Mach-O corpus, vs LC_FUNCTION_STARTS (2,056 truth) TP 1,787 -> 1,818, FP 1,113 -> 1,107 recall 86.916% -> 88.424%, no sample losing recall Two headline figures are marked as the contributor's own because they cannot be reproduced here at all: the `bti j` 803/0 split, since no bundled fixture holds a single `bti j` word, and #312's ARM64 PE result, since a scan of all 104 fixture files finds 8 i386 PEs, 4 AMD64, one ReadyToRun image and no `0xAA64`. Saying which figures are ours and which are theirs is more useful than a uniform tone of confidence. `Turtle_5f9cd91d8d1d`'s 12 dropped starts are attributed rather than left as an unexplained delta: they bisect to #307's inbound-call fix and fall inside the 175 false positives that change already measured, and the sample carries no LC_FUNCTION_STARTS to score them against individually. Closes #319 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr * chore: record binweight's serialized type change in the v4.5.1 entry Left open as a question in yesterday's review and resolved as "yes, it belongs in the entry". #302 moved `SmdaFunction.binweight`'s class default from int `0` to float `0.0` while clearing ty 0.0.74's diagnostics, and `binweight` is serialized through `toDict()`. The blast radius is small but real: the per-block accumulation already adds `float(...)`, so every function with at least one block was a float before this release too. Only a function with no blocks -- a zero-function or error report -- keeps the class default, and that value now writes as `0.0` where it wrote `0`. Verified on both trees rather than reasoned about: `SmdaFunction.binweight` is `0` at 6cc5aca and `0.0` at d111548. MCRIT stores these reports, so a consumer diffing them byte-for-byte will see it even though nothing reads the field as an integer. That is exactly the kind of change that costs someone an afternoon if it is not written down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr * chore: fold the two remaining #317 items into the v4.5.1 entry #320 and #321 finish the review pass that #318 started, so the release entry should carry all seven of its corrections rather than five. Both are behaviour-neutral typing work, but the Housekeeping text says what each one actually decided rather than listing them as annotations: `blocks` being typed forces the PIC/OPC hash path to refuse an instruction with no bytes instead of blanking it, and `extract_strings` could not be narrowed until three helpers and three `SmdaReport` attributes were declared first. Behaviour-neutrality is stated with the evidence rather than as a claim -- the corpus benchmark ran on #320 and reports 0 of 155 files differing across 175,946 functions, which is a stronger statement than the bundled fixtures can make. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr * chore: format the v4.5.1 entry as a nested list @r0ny123 asked on #319 whether the changelog could be formatted as a list to make it easier to read. At ~2,700 words on a single line, this entry was the one that provoked the question, so it gets the treatment now rather than waiting for the wider keep-a-changelog adoption tracked in #323. Deliberately narrow: only the v4.5.1 entry changes shape. The other 192 entries keep the one-line format, because converting them is a large, low-value rewrite that loses nuance in translation, and #323 proposes preserving them verbatim under an `Older releases` heading instead. Structure is the release summary as the top-level bullet, one sub-bullet per section, and a further level inside the three sections that cover more than one change -- Recovery (intel) splits into the switch-table fix, the alignment cut and the exception-directory refusal; Recovery (AArch64) into the inbound-call fix, the BTI work and the candidate-quality follow-ups; Housekeeping into the `ty` bump, the `ruff` pin, the five #318 corrections, the `binweight` serialization note and the two #317 typing items. The text is unchanged. Verified mechanically rather than by eye: stripping list markers and collapsing whitespace gives a string identical to the original entry. The two exceptions are disclosed rather than silent -- the first sub-bullet of Recovery (intel) and of Recovery (AArch64) had their opening letter capitalised, because splitting the section header onto its own line left them starting a bullet mid-sentence. Housekeeping's opens on a code span and needed nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
danielplohmann
added a commit
that referenced
this pull request
Sep 10, 2026
Three review findings on #300, all in text or in one expression, fixed here rather than sent back for another round. The config comments for USE_LSDA_LANDING_PADS and USE_ELF_FDE_INTERIOR_GAPS carried the per-rule figures measured before #304/#307/#309/#310/#311 landed -- the same baseline the PR description retracted, because master's own PPV on that corpus had moved from 76.676 to 91.497 while the branch waited. RESOLVE_TAILCALLS likewise still described the AArch64 bl fall-through gate as it measured before #307 changed what addTailcallCandidate does, and omitted the AArch64 ELF corpus entirely, which is the one row where the gate is a trade. All three now carry the re-measured attribution against the tree #300 landed on, and the tailcall block says what the 53 lost true positives are and how USE_ELF_EH_FRAME_CANDIDATES repays them. A description records a conversation; a config comment is what the next reader has. getExceptionDirectory matched the data directory with "EXCEPTION" not in str(directory.type), a substring of the enum's repr that would also claim any future type whose name contains it. The typed lookup is already the idiom in two other places, one of them the AArch64 walk over the same directory. The aarch64_static baseline moved by two starts and the test comment explained one of them. 0x40DF34 is refused, but it was not recovered before either, so the pair the fixture actually lost is 0x400350 and 0x40DF30 -- both mid-function instructions inside a declared FDE, which is a plainer reading than the Binary Ninja disagreement the comment led with. 0x400350 is now asserted alongside the other two. Also removes a comment left dangling in intel/FunctionCandidateManager.__init__ by #312, which hoisted the attribute it documented into common/ where the same text already sits. Suite 2060 passed, 1 skipped, 2593 subtests; ruff clean; ty exit 0 with 0 error-level diagnostics. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A function reached by falling through GCC's inter-function padding was decoded as a continuation of its predecessor, and the two came back as one function.
Root cause
The alignment cut had exactly one way in:
The theory behind that is that padding follows a function that ended in a noreturn call. GCC pads between functions whatever the previous one ended with — a
ret, ajmp, a tail call — so everything reached by falling through such padding was invisible to the cut.Fix
The other way in is the padding itself.
_paddingFillsToAlignmentreads the run whole: every byte from the current address to the next 16-byte boundary has to be alignment filler, and the run has to stop at that boundary.Reading the run rather than just its first encoding is what carries the change. Of the addresses it refuses that a bare "not on a 16-byte boundary" test would have cut, 1,598 of 1,778 are not functions:
That is still not enough on its own. GCC aligns loop heads with the same encodings it pads between functions with, and both runs end on the boundary, so the run test cannot separate them. So padding reached by falling through also needs its seed to decode as a function entry, on every format — the entry-shape gate that was PE-only becomes format-independent for this path. The call path keeps the old exemption, because the call is the prior that the previous function ended.
Without that second condition the bundled ELF fixtures gained five "functions" inside one unrolled string routine (
add rax, 4; mov cl, byte ptr [rax]; …), all 16-byte aligned, all with one inbound edge — loop heads, caught bytestPaddedLandingPad's control case and by the pinned fixture counts.Measured
296,225 ground-truth functions over 191 samples — gcc/clang/mingw C/C++, Go and Rust, 8 optimisation and link configurations each — exact function-start match:
+3,009 true starts, −2,676 false ones. Two true starts lost in total, both on C/C++; Rust loses none. All 47 Go cells are bit-identical, which is what an x86 padding rule ought to do to images whose pclntab already names every function.
An earlier variant without the entry-shape gate was better on Rust recall (+346 vs +246) but worse on Rust precision (84.82% vs 85.34%) and lost 12 true starts there. The gated version improves both metrics on the two corpora it moves and costs nothing on C/C++ or Go — those are bit-identical between the two variants, all 167 cells.
Where the baseline comes from. These figures are a paired base-vs-branch run where the base is the branch's own parent, so the deltas are this change's alone. That parent carried the work now open as #299 and #300 plus the AArch64 changes stacked after them, none of which touch this path — but the absolute PPV/TPR columns are that tree's, not plain master's, and the honest way to read the table is as the delta rather than as two absolute numbers.
Malware corpus
Against the 158-sample corpus the Malpedia Benchmark uses, classified with the repo's own
evaluate_runtime.pylabeller:Two files change by one address each, nothing dropped. That corpus is not reachable from a fork PR —
Evaluate & ReportandMalpedia Benchmarkboth report skipped here — so the run above is a local one, recorded because the comparison cannot happen in CI on this PR.Cost
Analysis time +3.4%, from a paired run (base/PR/PR/base, min-of-2, one process at a time) over ten samples of that corpus: 10.02s → 10.36s. The gate warns at 20% and fails at 30%.
Two things keep it there: the predicate returns before touching the buffer when the address is already on a boundary, and it rejects on the first byte via the existing
GAP_SEQUENCE_FIRST_BYTESset before scanning the run at all. Without that first-byte rejection the same measurement is +7.2%.MAX_GAP_SEQUENCE_LENGTHis derived once at import instead of per scan step.Also in here
GAP_SEQUENCESgains the two CS-prefixed nops binutils emits for widths 5 and 8. It already had the0x90-prefixed forms of both. Unlike the REX-prefixed sequences the suite explicitly bans from the shared table,0x2emeans the same thing in both modes — and the new test asserts that rather than assuming it, decoding each in 32- and 64-bit capstone and checking it comes back as oneleaof the stated length.Tests
Eight new cases, six of which fail on the parent commit:
_paddingFillsToAlignment: a run that fills the line, a run assembled from several encodings, a lone nop inside real code, a run that carries on past the boundary, an address already on a boundary, and a window truncated at one.leain both modes.Validation
ruff check/ruff format --checkpass,ty checkexit 0, full suite green on this branch rebased onto current master (1824 passed, 1 skipped, 2584 subtests), diff coverage 100% on all 23 changed lines againstupstream/master, pinned fixture counts unchanged (bashlite 177, mirai 150).Independent of #299 and #300 — neither touches
X86Backend.analyzeFunctionor the gap-sequence table — and merges cleanly against both.The
tycommit at the tipThe last commit on this branch is #302's fix, cherry-picked unchanged.
Master fails
Code Qualityon its own right now: 572acd7 bumpedtyto 0.0.74, and its newunsound-return-statement/unsound-assignmentrules become errors under[tool.ty.rules] all = "error". That is 41 errors across 15 files, none of which this PR touches — so every PR opened against master inherits a red check that says nothing about its own diff.Carrying #302's commit here means this PR's CI reflects only this PR's work. It is the same commit, unmodified, so the moment #302 lands this one is empty and falls out on rebase — there is nothing to untangle later. If you would rather look at this branch without it, merge #302 first and I will drop it.
Checked against
ty0.0.74 itself rather than whatever an older local pin resolves to: on master it exits 1 with 41 errors, on this branch it exits 0 with none.