Repository navigation
decomp: data in C, checked byte for byte in the boot ELF and every level - #179
Merged
Merged
Conversation
The retail switch table at 0x1E8770 sends action 1 to the same place as action 0 (break), not to the action 3 arm. The code bytes were the same either way, and the preserved core_rdata blob hid the wrong table entry.
The .rodata, .sdata and .data of the units in RODATA_OVERLAYS, SDATA_OVERLAYS and DATA_OVERLAYS used to be linked on top of the preserved blobs, and the blob bytes won. A wrong value in C still gave PASS, and data in a unit no table listed was dropped without a word. Each entry now also gives its size. The blobs are linked as the pieces between the C-owned ranges, so the gate hashes what the C compiled to. The link fails when a C section is not its configured size, when an entry names no configured unit, and when a unit defines data no entry places. render_level_effects_and_screen_sprites gets back the entry for its panel slot table, left under the unit's old name.
A D_LNN_XXXXXXXX object defined in src/overlays is compared with what level NN loads at XXXXXXXX, every pointer in it resolved to that level's address of its symbol (check_data, overlay-data-bytes-v1). make overlays runs it for every data object in C, fails on data without a D_LNN label, and checks config/overlays/us/data.tsv lists each one with its size.
total_data is the executable's preserved data and each level's .lit, .data and vtables; matched_data is the data C defines and a proof checks: the *_OVERLAYS entries (the boot gate) and config/overlays/us/data.tsv (make overlays). bss is not counted.
BSS_OVERLAYS (VMA, size) and SBSS_OVERLAYS (VMA, offset, size) place a
unit's uninitialised variables; the preserved bss is linked as the pieces
around them and drops the splat labels the unit now defines. bss has no
file bytes, so the gate also checks that every D_/FUN_/jtbl_ label is
linked at its own address. Common symbols link with their unit's bss.
data-refs.py --emit now covers every boot data section (const for
core_rdata, section(".bss") for bss) and, with --level N, level data:
values from the level's records, pointers named after the level's
functions and data, and the row added to config/overlays/us/data.tsv.
A level's small bss sits in its .lit as zeros; the data proof accepts it.
data-refs --emit now prints the shortest exact float, one row per line for nested arrays, and turns an extern in the target file into the definition.
snd_current_buffer, frm_clock_time and vif_chain_flags were declared with another type than their definition; D_0015EC80 also sat in undefined_syms.txt, which overrides the C definition.
Every D_ label now has one declaration with one name: plain local externs that repeat a header or another file are removed and their uses renamed. A label used by several files is declared once in include/rnc/data_labels.h (boot) or include/rnc/overlay/level_labels.h (levels) until its subsystem is known. Placements stay as they were: a file that reaches a label with sda, NOT_SDA or MACRO_ADDR keeps one alias for it, named <name>_gp, _lui or _macro when the plain name is also visible. A function that reads a label through two names keeps both (the second name is a lever). Labels whose declarations disagree on the type are left for the next step.
scripts/data-decls.py lists the D_ labels that src/ and include/ declare with more than one type or name (placement aliases _gp, _lui and _macro count as one name). make check fails when the count grows; it is lowered as each label is typed by hand.
Semaphore ids (sceCd, sceMc) are s32: they are compared with -1 and < 0. create_moby compares spawn_frame with the frame counter unsigned, so it casts there instead of changing the counter's type.
002deee0.c keeps its list: removing it changes the unit's code.
Each file's alias takes the label's real name, plus _gp, _lui or _macro when the file needs another placement.
…on past a failed address
…in all levels and counted in each
…jects) The proof resolves a reference to data the file itself defines (the assembler points at the section, gp-relative offsets minus gp0).
…once and proven in all levels
Hero step tables are int[4] in .sdata and the unk3E4 pointee is a HeroVelocityKey; l16/l17 vertex count and vertex pointer tables hold their counts and point at the named level data; offset vectors read by index are float[4].
Hero step tables (0015F6D8, L05 0015F6E8) are int[4] in .sdata, the
0015F6E8 pointee is the char[4] the shared hero code declares, the l02
shorts read as ints are ints, the l02 offset vector is float[4], and the
l08 vertex/colour tables point at their named level data. Scalars drop the
{0} braces; l07 scroll phase no longer shadows its local.
…able in C caption
…keeps D_ labels only
# Conflicts: # configure.py # src/textbin/world/data/select_world_object_resource_tables.c
Decompilation progress
No function changed its match state compared with main (ce39091). Compared with main (ce39091). C_FUZZY uses main's last measurement. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before
Most game data (tables, constants, strings, level data) was raw bytes in the preserved blobs, reached through
D_labels declared ad hoc in each file with mixed types. The level data that was already in C was linked under the retail blob, so nothing checked it: for example, a wrong switch table infun_0021abf8passed. The report counted only code.After
RODATA_OVERLAYS,SDATA_OVERLAYSandDATA_OVERLAYSnow carry(vma, offset, size). The link places the unit's section in place of the blob bytes and fails when the size differs, so the boot ELF hash covers the C data.verify-baseline.shfails when a label is linked away from its address.D_LNN_XXXXXXXXobject defined in C has a row inconfig/overlays/us/data.tsv.make overlayscompares its bytes, with pointers resolved, to what the level loads there (check_datainscripts/overlay_proof.py).D_XXXXXXXXrows (src/overlays/shared/data/level_globals.c) are checked in every level.struct HeroVelocityKey, whichhero_integrate_velocityreads.scripts/data-refs.pycatalogues the data the code touches (--catalog). It writes a definition plus its*_OVERLAYSentry ordata.tsvrow (--emit ADDR --into FILE [--level N]).scripts/data-decls.pylists labels whose declarations still disagree, ratcheted intest_public_tools.py.include/rnc/data_labels.hinto subsystem headers (include/rnc/<subsystem>/...). Globals several level files share, whose meaning is unknown, are listed by address ininclude/rnc/overlay/level_labels.h.report.jsonanddecomp_map.jsoncount data (total_data,matched_data, adataobject per part).bytes_matching_c_zerosays how much of the matched data is zeros: most of the overlay figure is large zero-filled level tables.CONTRIBUTING.md,docs/code-cleanup.md,docs/overlays.mdanddocs/progress-metrics.mdcover the steps, naming and placement rules, the proof and the metric..rodatasizes forprintf_impland_sceFs_Rcv_Intrare measured, and main'sactive_decode_buffername is kept.Verification
make elfends withPASS: reconstructed boot ELF matches retailmake overlaysends withPASS: all 2809 overlay functions in C match retail, with their 17298 copies in other places, and all 3895 level data objects in Cmake checkpasses (99 tool tests, script parse, report, map, layout checks)