fix(aarch64): read the BTI target type, and resume past a refused pad's block

[r0ny123]

Two commits' worth of one thing: the AArch64 candidate scans read a bti j landing pad as a function entry, and refusing one has to resume in the right place or the false positive simply moves four bytes.

The measurement first

Every prologue-sole booking on a 72-cell built AArch64 ELF corpus, charged to the arm of is_function_prologue that actually accepted it — same order the function uses, so each address is charged to the branch that took it:

pattern	FP	TP	FP share	FP per TP
BTI landing pad	803	505	59.8%	1.59
stp x29,x30,[sp,#imm]! frame record	536	2,764	39.9%	0.19
PAC sign (paciasp/pacibsp)	0	1,656	0.0%	0.00
not a prologue word	4	0	0.3%	—

The two remaining arms — the callee-saved pair stp x{19..28},x{19..28},[sp,#-imm]! and the LR save str x30,[sp,#-imm]! — are the sole booker of nothing at all here, neither a true positive nor a false one.

Splitting the BTI row by which form the compiler wrote:

variant	FP	TP
bti c (call target)	0	505
bti j (jump target)	803	0

Exact, not statistical. 1,308 addresses, no overlap, across 72 cells.

Why the two forms mean different things

bti j permits a target reached by br — an indirect jump, which is what a switch dispatch does to a case block. bti c permits one reached by blr — an indirect call, which is how a function is reached. A call landing on a J-only pad faults. So the compiler that emitted bti j was naming an interior label, and it recorded that fact in the instruction.

Candidate discovery read all four forms as entry prologues, which put every jump pad in the image on the list. The existing guard reads the word before the pad and refuses one that follows ordinary code — but that cannot separate these two, because a case block is preceded by the previous case's terminating branch, which is exactly the boundary shape the guard accepts as an entry.

So the pass had the answer in front of it and was not reading it.

Commit 1: read the target type

USE_AARCH64_BTI_TARGET_TYPE, on by default. A bti j word is interior by what it says about itself, before anything around it is read.

It lives in _isLikelyInteriorBtiCandidate, the predicate the prologue scan and the gap scan share, rather than in the scan that produced the histogram. Both reach these addresses — the pads that became false positives are the ones behind an indirect branch nothing resolves, so no analysis covers them and both passes get a turn. A rule in one would have left the other booking exactly what the first refused.

corpus	n	PPV	TPR	Δ FP	Δ TP
Built C/C++ AArch64 ELF	72	87.596 → 89.186	96.205 → 96.205	−2,941	−1

Recall unchanged to three decimals. 2,941 removed against 803 prologue-sole bookings, because a wrong entry does not only cost its own address — it starts an analysis that fragments the routine it sits inside.

The one true positive that moves is not lost

On one cell the entry at 0xe1744 is returned at 0xe1748 instead: one instruction later, at the stp x29, x30, [sp, #-96]! rather than at the paciasp in front of it. Both are prologue candidates, and this change shifts their order in the queue; the stp is analysed first and the paciasp behind it then aborts on the collision.

Read against truth rather than against the count, the window is better after, not worse:

in [0xe1500, 0xe1f00], truth declares 3 functions	recovered	of which real	spurious
before	13	3	10
after	6	2	4

An exact-start metric charges that as a lost function plus a false positive. It is a boundary that moved by four bytes, not a routine that vanished.

Commit 2: resume past the block, not past the pad

Refusing a pad advanced the gap scan by one instruction. That puts it on the pad's own first body instruction — ordinary code, which passes every remaining guard and is promoted in the pad's place. The false positive does not go, it moves four bytes:

	recovered functions
bti j, rule off	0x401000, 0x401008
bti j, rule on, resuming one instruction on	0x401000, 0x40100c
bti j, rule on, resuming past the block	0x401000

So the walk resumes past the block the pad labels — its first terminator — instead of past the pad.

Terminators are ret, br, b and the traps brk / hlt / udf. The traps matter because AArch64Backend.analyzeInstruction ends a function on all three through END_INS: without them the walk reads a block that has already closed and skips whatever lies behind it, and what lies behind a trap is exactly the unreferenced, gap-only routine this scan exists to reach. Reproduced on a five-word image — the walk resumed at 0x401018 instead of 0x40100c, swallowing a paciasp-opening routine, for each of the three trap forms.

It falls back to the single-instruction step when no terminator is within _GAP_RUN_LIMIT, so a run of undecodable bytes cannot make the skip travel an arbitrary distance. That bound is not hypothetical caution — an unbounded skip on a bad extent is what cost 1,274 functions on one dump while developing the x64 .pdata suppression, and the same shape is available here.

Applied to every arm of _isLikelyInteriorBtiCandidate, not only the target-type arm the first commit adds. The two older arms — a pad inside already-claimed code, and a pad following ordinary code — have the same weakness and predate that change. _GAP_RUN_LIMIT also replaces the bare 0x400 in _gapRunFlowsIntoInterior, which walks the same runs under the same bound and is the only other place that number appears.

corpus	n	PPV	TPR	Δ FP	Δ TP
Built C/C++ AArch64 ELF	72	89.215 → 89.623	96.208 → 96.208	−442	0
ARM64 Mach-O	11	bit-identical — 93.992 / 95.619, 2,484 TP, 262 FP both sides

442 false positives removed, recall unchanged to three decimals, and not one true positive moved — TP is 59,012 on both sides.

Why the first commit's tests did not catch it

They asserted the refused address was absent:

self.assertNotIn(BASE + 0x8, self._btiPadOnly(0xD503249F))

which is true both when the false positive is gone and when it has moved to BASE + 0xc. The assertion was satisfied by the exact defect it was written to prevent. All four BTI cases now compare the whole recovered set, which is what makes the difference between "removed" and "moved" visible.

Controls

corpus	n	result
Built C/C++, x86-64	140	bit-identical — 93.740 / 96.968, 113,438 TP, 14,123 FP both sides
Built Go, all platforms	47	bit-identical — 94.036 / 99.367, 165,627 TP, 10,944 FP both sides
ARM64 Mach-O	11	bit-identical for commit 1 too — 93.986 / 95.616, 2,484 TP, 263 FP both sides

The x86-64 corpus is the control that matters for scope: the intel scan has no BTI shape and the rule must not reach it.

The Mach-O corpus is not a control for the rule itself, and is not offered as one. Those 11 cells book no BTI-opened prologue candidate at all, so the rule is inert there rather than confirmed. It is measured on one corpus, and this says so.

The PE ARM64 exception-record reader (is_exception_record_entry) accepts the same word through a different question — whether a record names an independent entry or a funclet — and is deliberately left alone. There is no ARM64 PE corpus here to measure it on, and shipping an unmeasured suppression rule is the thing the .pdata work above argued against.

Tests

• Five cases in tests/testAArch64Disassembler.py over a landing pad reached only by an unresolved br — the shape that actually produced the false positives. A pad the engine can reach from inside a function is absorbed by the analysis whatever the prologue scan did, so it cannot show this rule doing anything; the first fixture written for this was that shape and it passed for the wrong reason. bti j refused; bti c and bti jc still booked in the same position; the flag off restores the old behaviour; and the predicate asserted directly, because the two callers differ only in which address they stand on.
• test_the_body_of_a_refused_pad_is_not_promoted_in_its_place — the walk landing past the block's ret.
• test_a_trap_ends_the_pad_run_like_any_other_terminator — all three trap forms as subtests.
• test_a_pad_run_with_no_terminator_in_reach_falls_back_to_one_step — the bound, over 0x200 words with no terminator.

Mutation-tested both ways: reverting the call site to += INSTRUCTION_SIZE fails test_a_jump_only_bti_pad_is_not_an_entry (which it did not do before the assertions were tightened), and deleting the trap clause fails all three subtests.

One follow-up deliberately not in here

_gapRunFlowsIntoInterior keeps a different terminator set from the skip walk — it walks to ret/br/b and reads straight through traps. That asymmetry is correct rather than an omission: the skip walk decides how far to skip, and one instruction too far steps over a real entry and loses it, so it must stop at the earliest boundary it can defend; the suppression walk decides whether to suppress, nothing is skipped either way, and reading further only gathers more evidence about the same candidate. Counted on both corpora, adding is_trap there releases 12 candidates on ARM64 Mach-O (two sites, swept one word at a time), 0 of them true starts, and 0 candidates on the 72-cell ELF corpus.

The test file that pins that reasoning needs a guard from #300 (_gapRunFlowsIntoInterior asking the live function set, not just the pre-analysis candidate snapshot), so it is held back rather than included here.

Validation

ruff check / ruff format --check pass, full suite green on this branch rebased onto current master, diff coverage 100% with 0 missing lines on both changed files.

Relationship to #299 and #300

Both commits are also sitting inside #299, and the first of the two inside #300 as well. Those were opened from branches that had my fork's master merged into them repeatedly, so their diffs are much wider than their titles suggest, they overlap each other, and they carry this change at two different points of its history.

This PR is the same change on its own, rebased onto current master, so it can be reviewed and measured for what it actually is. If you would rather take it through one of those, close this one; if you take this one, the matching hunks fall out of theirs on rebase.

The ty commit at the tip

The last commit on this branch is #302's fix, cherry-picked unchanged.

Master fails Code Quality on its own right now: 572acd7 bumped ty to 0.0.74, and its new unsound-return-statement / unsound-assignment rules become errors under [tool.ty.rules] all = "error". That is 41 errors across 15 files, none of which this PR touches — so every PR opened against master inherits a red check that says nothing about its own diff.

Carrying #302's commit here means this PR's CI reflects only this PR's work. It is the same commit, unmodified, so the moment #302 lands this one is empty and falls out on rebase — there is nothing to untangle later. If you would rather look at this branch without it, merge #302 first and I will drop it.

Checked against ty 0.0.74 itself rather than whatever an older local pin resolves to: on master it exits 1 with 41 errors, on this branch it exits 0 with none.