docs(docs): say which address space a report's offsets are in - #308
Merged
danielplohmann merged 3 commits intoSep 7, 2026
Merged
Conversation
A managed report's function and instruction offsets are file offsets. Every other backend reports virtual addresses. Nothing said so, so a consumer correlating a .NET report with a native one -- or with any other tool's output -- was comparing two different address spaces with no way to know. Documented rather than changed. The offsets are a public compatibility surface that downstream consumers index on, so moving them is a deliberate decision about the report format and not a bug fix. What was missing was the statement, and the fact that a report already names the backend that produced it twice -- in `architecture` and in the language score map -- so a consumer can tell which rule applies. Adds a test that pins it in both directions: every recovered method sits at its body's file offset, and none sits at the virtual address of the same method. The second is the control -- on this fixture the two spaces are far enough apart that only a report in the file's satisfies the first.
The first version of this said offsets are virtual addresses on "every backend but one". That is wrong, and wrong in the direction that matters: DalvikDisassembler.analyzeFunction addresses a method by its code item's offset and each instruction by that plus its position in the bytecode, so a Dalvik report is in the DEX file's space, not a virtual one. Documenting only CIL told a consumer to read Dalvik offsets as addresses -- the exact error the section exists to prevent. Verified rather than read: on the bundled DEX fixture, all 2219 recovered offsets are code-item offsets LIEF independently reports for the same file. The README now names all four backends in a table, and records why the two managed cases differ. A CIL method has an RVA and reporting the file offset instead is a choice; a DEX carries no load address at all, so the file offset is the only address there is. Adding base_addr to either produces a number that means nothing, which is what a consumer needs to know. The Dalvik test cross-checks against LIEF rather than against the file's length: with base_addr at 0 an address inside the image proves nothing about which space it is in.
Keep the 0.0.74 bump and satisfy the new unsound-return/assignment/yield rules with annotations on the public DTO and helper surfaces. Ignore ty in Dependabot so the next patch release cannot redden master unattended.
This was referenced Sep 7, 2026
Merged
This was referenced Sep 8, 2026
danielplohmann
added a commit
that referenced
this pull request
Sep 8, 2026
* chore: draft the v4.5.1 release notes The changelog entry for what has landed since v4.5.0 (6cc5aca): #302, #303, #304, #305, #306, #307, #308, #310 and #316, plus the Dependabot ty bump 572acd7 that #302 had to answer for. Held as a draft rather than a release. The version is deliberately NOT bumped here -- #309, #311 and #312 are reviewed and wanted but waiting on a rebase, and splitting one coherent accuracy pass across two releases makes both entries weaker. What this commit does is stop the writeup living in a scratchpad while that happens. To finish it: add sections for #309, #311 and #312, bump VERSION in src/smda/SmdaConfig.py and __version__ in src/smda/__init__.py, and move the date to the actual release day. No escaper output changed anywhere in this set -- the only edit to intel/definitions.py adds two GAP_SEQUENCES entries, which feed padding detection and not escaping, and testEscaperFingerprint passes unchanged -- so ESCAPER_DOWNWARD_COMPATIBILITY stays at 4.4.5 and INTEL_PIC_HASH_ESCAPE_VERSION at 4.3.5, and no report needs reprocessing. Figures are the contributors' own except where this session reproduced them, and the two that were reproduced are stated as measured here: the padding cut's analysis-time cost (+3.4% claimed, +3.84% measured) and the fixture movement on rust_pe_gnu_xored (+26 real starts against +8 false, scored against that fixture's own COFF symbol table rather than the corpus macro mean). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: fold #318 into the v4.5.1 release notes #318 landed as 395c88d after this entry was written. Its five corrections go under Housekeeping: the README's metadata.language claim, the restored `-> bool`, the ruff-check hook rename, the GAP_SEQUENCES reach note, and the docstring on the queue-rebuild test. Keeping this current as things merge is the point of the branch existing, so it does not go stale between now and the release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: cut v4.5.1 Finishes the draft now that #309, #311 and #312 have landed. Adds their sections, bumps `VERSION` and `__version__` to 4.5.1, and moves the date to the release day. The three new entries: #309's exception-directory refusal of interior gap candidates on x64, with the chained-versus-primary distinction and the seeding-is-not-suppression principle that keeps carved records out of it; #311's four candidate-quality defects, including the corrected entry at `0x40df30` on `aarch64_static_xored` and the metadata-coverage gate; and #312's ARM64 counterpart, which reconstructs the extent an ARM64 `RUNTIME_FUNCTION` does not carry and hoists the shared lookup into `common/`. `USE_MACHO_ADDRESS_REF_CANDIDATES` gets its own **Defaults changed** section. It is the only default whose value moves in this release, it changes recovery output on Mach-O input with no action by the caller, and a default going from off to on inside a follow-ups PR is exactly the kind of thing that gets lost. The closing movement section is rewritten to describe v4.5.0 -> v4.5.1 rather than any single PR, and both figures in it were measured on this tree against independent truth rather than quoted: rust_pe_gnu_xored, vs its 2,186 retained COFF .text function symbols 2,355 -> 2,201 starts, +26 real / -180 false recall 92.452% -> 93.641% precision 85.817% -> 93.003% 11-sample ARM64 Mach-O corpus, vs LC_FUNCTION_STARTS (2,056 truth) TP 1,787 -> 1,818, FP 1,113 -> 1,107 recall 86.916% -> 88.424%, no sample losing recall Two headline figures are marked as the contributor's own because they cannot be reproduced here at all: the `bti j` 803/0 split, since no bundled fixture holds a single `bti j` word, and #312's ARM64 PE result, since a scan of all 104 fixture files finds 8 i386 PEs, 4 AMD64, one ReadyToRun image and no `0xAA64`. Saying which figures are ours and which are theirs is more useful than a uniform tone of confidence. `Turtle_5f9cd91d8d1d`'s 12 dropped starts are attributed rather than left as an unexplained delta: they bisect to #307's inbound-call fix and fall inside the 175 false positives that change already measured, and the sample carries no LC_FUNCTION_STARTS to score them against individually. Closes #319 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr * chore: record binweight's serialized type change in the v4.5.1 entry Left open as a question in yesterday's review and resolved as "yes, it belongs in the entry". #302 moved `SmdaFunction.binweight`'s class default from int `0` to float `0.0` while clearing ty 0.0.74's diagnostics, and `binweight` is serialized through `toDict()`. The blast radius is small but real: the per-block accumulation already adds `float(...)`, so every function with at least one block was a float before this release too. Only a function with no blocks -- a zero-function or error report -- keeps the class default, and that value now writes as `0.0` where it wrote `0`. Verified on both trees rather than reasoned about: `SmdaFunction.binweight` is `0` at 6cc5aca and `0.0` at d111548. MCRIT stores these reports, so a consumer diffing them byte-for-byte will see it even though nothing reads the field as an integer. That is exactly the kind of change that costs someone an afternoon if it is not written down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr * chore: fold the two remaining #317 items into the v4.5.1 entry #320 and #321 finish the review pass that #318 started, so the release entry should carry all seven of its corrections rather than five. Both are behaviour-neutral typing work, but the Housekeeping text says what each one actually decided rather than listing them as annotations: `blocks` being typed forces the PIC/OPC hash path to refuse an instruction with no bytes instead of blanking it, and `extract_strings` could not be narrowed until three helpers and three `SmdaReport` attributes were declared first. Behaviour-neutrality is stated with the evidence rather than as a claim -- the corpus benchmark ran on #320 and reports 0 of 155 files differing across 175,946 functions, which is a stronger statement than the bundled fixtures can make. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr * chore: format the v4.5.1 entry as a nested list @r0ny123 asked on #319 whether the changelog could be formatted as a list to make it easier to read. At ~2,700 words on a single line, this entry was the one that provoked the question, so it gets the treatment now rather than waiting for the wider keep-a-changelog adoption tracked in #323. Deliberately narrow: only the v4.5.1 entry changes shape. The other 192 entries keep the one-line format, because converting them is a large, low-value rewrite that loses nuance in translation, and #323 proposes preserving them verbatim under an `Older releases` heading instead. Structure is the release summary as the top-level bullet, one sub-bullet per section, and a further level inside the three sections that cover more than one change -- Recovery (intel) splits into the switch-table fix, the alignment cut and the exception-directory refusal; Recovery (AArch64) into the inbound-call fix, the BTI work and the candidate-quality follow-ups; Housekeeping into the `ty` bump, the `ruff` pin, the five #318 corrections, the `binweight` serialization note and the two #317 typing items. The text is unchanged. Verified mechanically rather than by eye: stripping list markers and collapsing whitespace gives a string identical to the original entry. The two exceptions are disclosed rather than silent -- the first sub-bullet of Recovery (intel) and of Recovery (AArch64) had their opening letter capitalised, because splitting the section header onto its own line left them starting a bullet mid-sentence. Housekeeping's opens on a code span and needed nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDCLUmqGzE5fwJqUWsSStr --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A managed report's function and instruction offsets are file offsets. The native backends report virtual addresses. Nothing in the repository said so, so a consumer correlating a managed report with a native one — or with
base_addrplus an RVA, or with any other tool's output — was silently comparing two different address spaces.Two backends are affected, not one:
report.architectureintel,aarch64base_addrplus an RVAcildalvikThe two managed cases differ in why, which is worth recording. A CIL method has an RVA and
CilDisassemblerreports the file offset instead — that is a choice. A DEX carries no load address at all, so for Dalvik the file offset is the only address there is. The consequence for a consumer is identical either way: addingbase_addrto one of them produces a number that means nothing.The decision: document it, do not move it
The offsets are a public compatibility surface that downstream consumers index on. Converting them would change every stored managed report's addresses; carrying both would change the report schema. Either is a deliberate decision about the format rather than a bug fix, and neither belongs in a change that can be made without the people who consume it. What was actually missing is the statement.
The second half of the statement matters as much as the first: a report already names the backend that produced it, twice —
report.architecture, andmetadata.languageintoDict(). So a consumer has what it needs to apply the right rule, once it knows there are two.What changed
READMEsubsection under Usage, beside the raw-buffer section that covers the other thing about a report you cannot infer from it.CilDisassemblerandDalvikDisassembler, at the places the conversion is actually made.tests/testCilAddressSpace.py.No behaviour change. The only edits under
src/smda/**are docstrings.The tests
Both halves pin the contract against a second parser, not against the file's length — with
base_addrat 0 an address inside the image proves nothing about which space it is in, and that weaker assertion would pass on a report that had been converted to virtual addresses.MethodDef.Rvathroughget_offset_from_rva), and none sits at the virtual address of the same method (ImageBase + Rva). The second is the control that makes the first mean something.code_offsetvalues LIEF independently reports for the same file.The Dalvik half is a correction to the first draft of this, which said offsets are virtual addresses on "every backend but one". That was wrong in the direction that matters: documenting only CIL would have told a consumer to read Dalvik offsets as addresses, the exact error the section exists to prevent.
Validation
ruff check/ruff format --checkpass, full suite green on this branch rebased onto current master (1820 passed, 1 skipped, 2584 subtests). Diff coverage reports no measurable lines, because the onlysrc/smda/**lines are docstrings.Relationship to #299 and #300
This change is also sitting inside #299 and #300. Both of those were opened from branches that had my fork's master merged into them repeatedly, so their diffs are much wider than their titles suggest and they overlap each other as well.
This PR is the same change on its own, rebased onto current master, so it can be reviewed and measured for what it actually is. If you would rather take it through one of those, close this one; if you take this one, the matching hunks fall out of theirs on rebase.
The
tycommit at the tipThe last commit on this branch is #302's fix, cherry-picked unchanged.
Master fails
Code Qualityon its own right now: 572acd7 bumpedtyto 0.0.74, and its newunsound-return-statement/unsound-assignmentrules become errors under[tool.ty.rules] all = "error". That is 41 errors across 15 files, none of which this PR touches — so every PR opened against master inherits a red check that says nothing about its own diff.Carrying #302's commit here means this PR's CI reflects only this PR's work. It is the same commit, unmodified, so the moment #302 lands this one is empty and falls out on rebase — there is nothing to untangle later. If you would rather look at this branch without it, merge #302 first and I will drop it.
Checked against
ty0.0.74 itself rather than whatever an older local pin resolves to: on master it exits 1 with 41 errors, on this branch it exits 0 with none.