Skip to content

fix(intel): keep the switch index tied across a relative dispatch's base add - #306

Merged
danielplohmann merged 3 commits into
danielplohmann:masterfrom
r0ny123:upstream-intel-switch-index-tie
Sep 7, 2026
Merged

fix(intel): keep the switch index tied across a relative dispatch's base add#306
danielplohmann merged 3 commits into
danielplohmann:masterfrom
r0ny123:upstream-intel-switch-index-tie

Conversation

@r0ny123

@r0ny123 r0ny123 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

A switch inside brotli built by gcc at -O2 has 27 cases. SMDA resolves 2 of them, queues those 2 as blocks of the function, and leaves the other 25 case bodies unreferenced — so the gap scan finds them, sees a common function start, and books five of them as functions of their own, inside the function they belong to.

That is a false positive and a shattered function in one move, and it happens on every dispatch of this shape.

Root cause

A 64-bit relative switch ends by adding the table's base to the entry the table read produced, and the branch reads that sum:

    lea     rsi, [rip + 0x6e189]        ; table base
    mov     qword ptr [rsp], rsi        ; spilled, the dispatch is reached from several places
    cmp     dword ptr [rbx], 0x1a       ; the real bound, checked against a memory cell
    ja      default
    mov     rdi, qword ptr [rsp]        ; reloaded
    mov     eax, dword ptr [rbx]
    movsxd  rax, dword ptr [rdi + rax*4]
    add     rax, rdi
    notrack jmp rax

_findJumpTableSize starts at the branch register and follows it backwards through copies, so it ought to reach movsxd, take rax out of the scaled operand, follow that back through mov eax, dword ptr [rbx] to the cell, and return 27 from the compare against it. That is the whole point of the index tracking already in there.

It never got that far. add rax, rdi is not one of the copy mnemonics, so the walk fell into the "anything else redefines whatever it writes" arm and dropped the tie one instruction short of the table read. With nothing tied, the untied fallback answers with the first register-against-immediate compare in the window — here an unrelated compare from earlier in the function — and the table comes out sized at 2.

The bug is one step from the end of a walk that was otherwise correct, which is why this looked like the jump-table pass simply not following this shape.

Fix

The walk carries the tie across the step that adds a base to the value it overwrites, in both spellings a compiler uses: add rax, rdi (register or memory source), and a lea whose address reads the register it writes.

Only the instruction the branch reads is asked. That is where the combine sits in every shape the analyzer recognises — the four relative arms all match on it being the nearest instruction, and the pattern-C lea is in the same position. An add further back is index arithmetic: add rax, rcx before the table read makes the index a sum, and a compare bounding one summand is not a bound on it. (That narrowing is the second commit; a bound recovered too small is the worse direction, because it is trusted as exact and truncates the scan.)

Two things deliberately still redefine, and both have a test:

  • An immediate source (add rax, 8) is index arithmetic, not a base.
  • A write back into a memory cell (add dword ptr [rbx], eax) is not this step at all — the step recognised here ends in the register the branch reads.

lea rax, [rip + 0x2004] is how the base itself is loaded and derives nothing from rax, so it is excluded by the same rule that admits lea rcx, [r11 + rcx].

Measured

140 built C/C++ cells (gcc and clang, x86-64 ELF, 10 programs × 7 variants × 2 toolchains), 117,654 truth functions from the unstripped link's symbol table, exact-address match, macro means:

PPV TPR TP FP
before 93.740 96.968 113,438 14,123
after 93.765 96.968 113,438 14,103

20 false positives removed, no true positive lost on any cell, and 135 of the 140 cells come back bit-identical. The five that move:

cell TP FP
brotli_gcc-x64_O2 293 → 293 5 → 0
brotli_gcc-x64_O3 282 → 282 4 → 0
brotli_gcc-x64_O2-static 1245 → 1245 58 → 53
googletest_gcc-x64_O2-static 5820 → 5820 1793 → 1790
googletest_clang-x64_O2-static 5767 → 5767 1484 → 1481

The clang cell moves because a statically linked image pulls in gcc-built glibc.

The narrowing in the second commit changes no cell: the recovered function set is identical under both predicates, so this corpus does not contain the index-arithmetic shape and the narrowing costs nothing measured.

Second corpus, as a control on the shape. 8 built Rust cells (rustc 1.94, gnu targets, four profiles) are bit-identical — and not one dispatch on any of them changed its bound, checked by recording the recovered size at every dispatch under both trees. That is the expected result rather than a null one: LLVM leaves the bound check as the nearest register compare, so the untied fallback was already reading the right one. The shape this fixes is what gcc emits when the base is spilled across the bound check.

Bug-class sweep

The class is a value walk that reads a read-modify-write of the register it is chasing as a redefinition. Swept the other backward walks:

  • aarch64/analyzers.py jump-table resolution — already correct. It adds both source registers of its merging add back into the tracked set, and ties the bound to the index register taken from the table load rather than from the branch operand.
  • intel/IndirectCallAnalyzer.py and the syscall-number walk in intel/X86Backend.pynot instances. Both chase a concrete value rather than a provenance tie, so carrying either past an unmodelled write would report a wrong value. Stopping there is correct for them.
  • JumpTableAnalyzer._ripRelativeBase — same, and its docstring already calls the stop deliberate.

Tests

Eight new cases in tests/testJumpTableIndexBound.py:

  • the spilled-base dispatch recovers 27 where the fallback recovers 2 — with the fallback's answer asserted on the same window, so the assertion cannot pass on a bound the fallback happened to get right
  • lea that reads the register it writes carries the tie; lea that only loads an address does not
  • an immediate add, an add back into a tracked memory cell, and index arithmetic before the table read all still drop the tie, each with a positive control
  • ShatteredSwitchTestSuite builds a 64-bit image with exactly this dispatch and a decoy compare, and asserts no case body becomes a function of its own. Before the fix two of the four do. Its control asserts the three real functions are all recovered, so an empty report cannot satisfy it.

Run against the parent commit, the load-bearing assertions fail and the controls pass.

Validation

ruff check / ruff format --check pass, full suite green on this branch rebased onto current master (1824 passed, 1 skipped, 2584 subtests), diff coverage 100% on the changed lines.

On the malpedia corpus the function set moves on one file of 155: three fragments stop being functions and become blocks, and three real routines are recovered. That corpus is not reachable from a fork PR — Evaluate & Report and Malpedia Benchmark both report skipped here — so it is recorded rather than shown by CI. The bundled Fixture Benchmark Gate is green.

Relationship to #299 and #300

This change is also sitting inside #299 and #300. Both of those were opened from branches that had my fork's master merged into them repeatedly, so their diffs are much wider than their titles suggest and they overlap each other as well.

This PR is the same change on its own, rebased onto current master, so it can be reviewed and measured for what it actually is. If you would rather take it through one of those, close this one; if you take this one, the matching hunks fall out of theirs on rebase.

The ty commit at the tip

The last commit on this branch is #302's fix, cherry-picked unchanged.

Master fails Code Quality on its own right now: 572acd7 bumped ty to 0.0.74, and its new unsound-return-statement / unsound-assignment rules become errors under [tool.ty.rules] all = "error". That is 41 errors across 15 files, none of which this PR touches — so every PR opened against master inherits a red check that says nothing about its own diff.

Carrying #302's commit here means this PR's CI reflects only this PR's work. It is the same commit, unmodified, so the moment #302 lands this one is empty and falls out on rebase — there is nothing to untangle later. If you would rather look at this branch without it, merge #302 first and I will drop it.

Checked against ty 0.0.74 itself rather than whatever an older local pin resolves to: on master it exits 1 with 41 errors, on this branch it exits 0 with none.

…ase add

A 64-bit relative switch ends by adding the table's base to the entry the
table read produced, and the branch reads that sum:

    lea     rsi, [rip + 0x6e189]
    mov     qword ptr [rsp], rsi
    cmp     dword ptr [rbx], 0x1a
    ja      default
    mov     rdi, qword ptr [rsp]
    mov     eax, dword ptr [rbx]
    movsxd  rax, dword ptr [rdi + rax*4]
    add     rax, rdi
    notrack jmp rax

Bound recovery starts at the branch register and follows it back through
copies, so it should reach the table read, take the index out of the scaled
operand, and land on the compare that bounds it. It never got that far.
`add rax, rdi` is not a copy, so the walk read it as a redefinition and let
the tie go one instruction short of the table read. With nothing tied, the
fallback answers with the first `cmp <reg>, <imm>` in the window, which here
is an unrelated compare from earlier in the function.

On brotli built by gcc at -O2 that sized a 27-entry table at 2. The other 25
case bodies were never queued as blocks of the function that dispatches to
them, so the gap scan found them unreferenced and booked five of them as
functions of their own, inside the function they belong to.

The walk now carries the tie across the step that adds a base to the value
it overwrites, in both spellings a compiler uses for it: `add <reg>, <reg or
memory>`, and a `lea` whose address reads the register it writes. An
immediate source is index arithmetic rather than a base, and a write back
into a memory cell is not this step at all, so both still redefine.

Measured against compiler symbol tables on 140 built C/C++ cells (gcc and
clang, x86-64 ELF, 117,389 truth functions), macro means:

    PPV 93.630 -> 93.655 at TPR 96.962 -> 96.962

20 false positives removed, no true positive lost on any cell, and 135 of
the 140 cells come back bit-identical. 8 built Rust cells are bit-identical
too, and not one dispatch there changed its bound: LLVM leaves the bound
check as the nearest register compare, so the untied fallback already read
the right one.

Swept the class -- a value walk that reads a read-modify-write of the
register it is chasing as a redefinition. The AArch64 jump-table analyzer
adds both source registers of its merging add back into the tracked set, so
it never had this. The two other backward walks, the indirect-call resolver
and the syscall-number walk, chase a concrete value rather than a
provenance tie: carrying either past an unmodelled write would report a
wrong value, so stopping there is right and neither is an instance.
The base-add carry admitted any register or memory `add` writing a tracked
register, anywhere in the backtrack window. Index arithmetic looks the same:

    mov     eax, dword ptr [rbx]
    add     rax, rcx
    movsxd  rax, dword ptr [rdi + rax*4]
    add     rax, rdi
    jmp     rax

Here the index is a sum, and the compare against `[rbx]` bounds one summand
of it. Carrying the tie across `add rax, rcx` reported that compare as the
table's bound -- and a recovered bound is trusted as exact, so an undersized
one truncates the scan and leaves the later case bodies for the gap scan to
book as functions, which is the failure this branch set out to fix.

Only the instruction the branch reads is asked now. That is where the base
combine is in every shape the analyzer recognizes: the four relative arms
all match on it being the nearest instruction, and the pattern-C `lea` is
in the same position.

No cell of the 140-cell C/C++ corpus changes: the recovered function set is
identical under both predicates, so the narrowing costs nothing measured
and removes a case the corpus does not happen to contain.

Measured again on the repaired ground truth, macro means:

    before  PPV 93.740 at TPR 96.968, 113,438 TP, 14,123 FP
    after   PPV 93.765 at TPR 96.968, 113,438 TP, 14,103 FP

Same 20 false positives, same absence of any recall cost, as against the
truth this branch was first measured on.
Keep the 0.0.74 bump and satisfy the new unsound-return/assignment/yield
rules with annotations on the public DTO and helper surfaces. Ignore ty
in Dependabot so the next patch release cannot redden master unattended.
@danielplohmann
danielplohmann merged commit 8fee560 into danielplohmann:master Sep 7, 2026
26 checks passed
@danielplohmann danielplohmann mentioned this pull request Sep 7, 2026
@r0ny123
r0ny123 deleted the upstream-intel-switch-index-tie branch September 7, 2026 17:38
danielplohmann added a commit that referenced this pull request Sep 8, 2026
* chore: draft the v4.5.1 release notes

The changelog entry for what has landed since v4.5.0 (6cc5aca): #302, #303,
#304, #305, #306, #307, #308, #310 and #316, plus the Dependabot ty bump
572acd7 that #302 had to answer for.

Held as a draft rather than a release. The version is deliberately NOT
bumped here -- #309, #311 and #312 are reviewed and wanted but waiting on a
rebase, and splitting one coherent accuracy pass across two releases makes
both entries weaker. What this commit does is stop the writeup living in a
scratchpad while that happens.

To finish it: add sections for #309, #311 and #312, bump VERSION in
src/smda/SmdaConfig.py and __version__ in src/smda/__init__.py, and move the
date to the actual release day.

No escaper output changed anywhere in this set -- the only edit to
intel/definitions.py adds two GAP_SEQUENCES entries, which feed padding
detection and not escaping, and testEscaperFingerprint passes unchanged --
so ESCAPER_DOWNWARD_COMPATIBILITY stays at 4.4.5 and
INTEL_PIC_HASH_ESCAPE_VERSION at 4.3.5, and no report needs reprocessing.

Figures are the contributors' own except where this session reproduced them,
and the two that were reproduced are stated as measured here: the padding
cut's analysis-time cost (+3.4% claimed, +3.84% measured) and the fixture
movement on rust_pe_gnu_xored (+26 real starts against +8 false, scored
against that fixture's own COFF symbol table rather than the corpus macro
mean).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: fold #318 into the v4.5.1 release notes

#318 landed as 395c88d after this entry was written. Its five corrections go
under Housekeeping: the README's metadata.language claim, the restored
`-> bool`, the ruff-check hook rename, the GAP_SEQUENCES reach note, and the
docstring on the queue-rebuild test.

Keeping this current as things merge is the point of the branch existing, so
it does not go stale between now and the release.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: cut v4.5.1

Finishes the draft now that #309, #311 and #312 have landed. Adds their
sections, bumps `VERSION` and `__version__` to 4.5.1, and moves the date to the
release day.

The three new entries: #309's exception-directory refusal of interior gap
candidates on x64, with the chained-versus-primary distinction and the
seeding-is-not-suppression principle that keeps carved records out of it; #311's
four candidate-quality defects, including the corrected entry at `0x40df30` on
`aarch64_static_xored` and the metadata-coverage gate; and #312's ARM64
counterpart, which reconstructs the extent an ARM64 `RUNTIME_FUNCTION` does not
carry and hoists the shared lookup into `common/`.

`USE_MACHO_ADDRESS_REF_CANDIDATES` gets its own **Defaults changed** section. It
is the only default whose value moves in this release, it changes recovery
output on Mach-O input with no action by the caller, and a default going from
off to on inside a follow-ups PR is exactly the kind of thing that gets lost.

The closing movement section is rewritten to describe v4.5.0 -> v4.5.1 rather
than any single PR, and both figures in it were measured on this tree against
independent truth rather than quoted:

  rust_pe_gnu_xored, vs its 2,186 retained COFF .text function symbols
    2,355 -> 2,201 starts, +26 real / -180 false
    recall    92.452% -> 93.641%
    precision 85.817% -> 93.003%

  11-sample ARM64 Mach-O corpus, vs LC_FUNCTION_STARTS (2,056 truth)
    TP 1,787 -> 1,818, FP 1,113 -> 1,107
    recall 86.916% -> 88.424%, no sample losing recall

Two headline figures are marked as the contributor's own because they cannot be
reproduced here at all: the `bti j` 803/0 split, since no bundled fixture holds
a single `bti j` word, and #312's ARM64 PE result, since a scan of all 104
fixture files finds 8 i386 PEs, 4 AMD64, one ReadyToRun image and no `0xAA64`.
Saying which figures are ours and which are theirs is more useful than a uniform
tone of confidence.

`Turtle_5f9cd91d8d1d`'s 12 dropped starts are attributed rather than left as an
unexplained delta: they bisect to #307's inbound-call fix and fall inside the
175 false positives that change already measured, and the sample carries no
LC_FUNCTION_STARTS to score them against individually.

Closes #319

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr

* chore: record binweight's serialized type change in the v4.5.1 entry

Left open as a question in yesterday's review and resolved as "yes, it belongs
in the entry". #302 moved `SmdaFunction.binweight`'s class default from int `0`
to float `0.0` while clearing ty 0.0.74's diagnostics, and `binweight` is
serialized through `toDict()`.

The blast radius is small but real: the per-block accumulation already adds
`float(...)`, so every function with at least one block was a float before this
release too. Only a function with no blocks -- a zero-function or error report
-- keeps the class default, and that value now writes as `0.0` where it wrote
`0`. Verified on both trees rather than reasoned about: `SmdaFunction.binweight`
is `0` at 6cc5aca and `0.0` at d111548.

MCRIT stores these reports, so a consumer diffing them byte-for-byte will see
it even though nothing reads the field as an integer. That is exactly the kind
of change that costs someone an afternoon if it is not written down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr

* chore: fold the two remaining #317 items into the v4.5.1 entry

#320 and #321 finish the review pass that #318 started, so the release entry
should carry all seven of its corrections rather than five.

Both are behaviour-neutral typing work, but the Housekeeping text says what each
one actually decided rather than listing them as annotations: `blocks` being
typed forces the PIC/OPC hash path to refuse an instruction with no bytes
instead of blanking it, and `extract_strings` could not be narrowed until three
helpers and three `SmdaReport` attributes were declared first.

Behaviour-neutrality is stated with the evidence rather than as a claim -- the
corpus benchmark ran on #320 and reports 0 of 155 files differing across
175,946 functions, which is a stronger statement than the bundled fixtures can
make.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr

* chore: format the v4.5.1 entry as a nested list

@r0ny123 asked on #319 whether the changelog could be formatted as a list to
make it easier to read. At ~2,700 words on a single line, this entry was the one
that provoked the question, so it gets the treatment now rather than waiting for
the wider keep-a-changelog adoption tracked in #323.

Deliberately narrow: only the v4.5.1 entry changes shape. The other 192 entries
keep the one-line format, because converting them is a large, low-value rewrite
that loses nuance in translation, and #323 proposes preserving them verbatim
under an `Older releases` heading instead.

Structure is the release summary as the top-level bullet, one sub-bullet per
section, and a further level inside the three sections that cover more than one
change -- Recovery (intel) splits into the switch-table fix, the alignment cut
and the exception-directory refusal; Recovery (AArch64) into the inbound-call
fix, the BTI work and the candidate-quality follow-ups; Housekeeping into the
`ty` bump, the `ruff` pin, the five #318 corrections, the `binweight`
serialization note and the two #317 typing items.

The text is unchanged. Verified mechanically rather than by eye: stripping list
markers and collapsing whitespace gives a string identical to the original entry.
The two exceptions are disclosed rather than silent -- the first sub-bullet of
Recovery (intel) and of Recovery (AArch64) had their opening letter capitalised,
because splitting the section header onto its own line left them starting a
bullet mid-sentence. Housekeeping's opens on a code span and needed nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants