Metroid Prime hacking research

2016-02-19 07:03:04 pm

Awesome, nice work Jack.

Quote from jackoalan:

Only works with GM8E-0-00 (verified by hash) and samples are rounded down to the nearest even number.

So you hash the dol to verify the version? Doesn't the hash change after you modify one of the loop points or am I misunderstanding something?

jackoalan

Edit history:

jackoalan: 2016-02-19 08:09:28 pm

jackoalan: 2016-02-19 08:08:48 pm

jackoalan: 2016-02-19 08:07:43 pm

2016-02-19 07:34:38 pm

The hash-stream zeroes the dynamic sections on the fly:
https://github.com/jackoalan/RSFLoopPatcher/blob/master/main.c#L209

Although I'll add a -f flag to skip the check for those who know what they are doing (for patching a .dol with other patches already).

Edit: I just made a stealth-edit to .exe on releases page, you can now run as:

Code:

RSFLoopPatcher boot.dol -f

to skip the hash-check.. USE WITH CARE!!! IT WILL CORRUPT ANY OTHER FILE!!!

Tanks

2016-02-20 11:32:13 pm

Quote from jackoalan:

OK! Got the RSF loop-editor tool ready:
https://github.com/jackoalan/RSFLoopPatcher/releases

You can drag boot.dol on top of the executable to start it in cmd.exe, then follow the interactive prompts:

Code:

Validating 'boot.dol'
 4177152 bytes

1: frontend_1.rsf [416480, 1973664]
2: frontend_2.rsf [273556, 1636980]
3: ending3.rsf [0, 6126592]
4: samusjak.rsf [2718088, 7916384]
Type a number [1-4] to change, or 's' to save, or 'q' to cancel:
frontend_2.rsf start: 1234
frontend_2.rsf end: 5678

1: frontend_1.rsf [416480, 1973664]
2: frontend_2.rsf [1234, 5678]*
3: ending3.rsf [0, 6126592]
4: samusjak.rsf [2718088, 7916384]
Type a number [1-4] to change, or 's' to save, or 'q' to cancel:

Only works with GM8E-0-00 (verified by hash) and samples are rounded down to the nearest even number.

@parax: Thanks for the heads-up about the sign-extension, got it in just how you suggested

This is fantastic! Gonna take it for a spin tomorrow and I'll let you all know the results!

[quote="jackoalan"]OK! Got the RSF loop-editor tool ready:
https://github.com/jackoalan/RSFLoopPatcher/releases

You can drag boot.dol on top of the executable to start it in cmd.exe, then follow the interactive prompts:

[code]
Validating 'boot.dol'
 4177152 bytes

1: frontend_1.rsf [416480, 1973664]
2: frontend_2.rsf [273556, 1636980]
3: ending3.rsf [0, 6126592]
4: samusjak.rsf [2718088, 7916384]
Type a number [1-4] to change, or 's' to save, or 'q' to cancel:
frontend_2.rsf start: 1234
frontend_2.rsf end: 5678

1: frontend_1.rsf [416480, 1973664]
2: frontend_2.rsf [1234, 5678]*
3: ending3.rsf [0, 6126592]
4: samusjak.rsf [2718088, 7916384]
Type a number [1-4] to change, or 's' to save, or 'q' to cancel:
[/code]
Only works with GM8E-0-00 (verified by hash) and samples are rounded down to the nearest even number.

@parax: Thanks for the heads-up about the sign-extension, got it in just how you suggested[/quote]
This is fantastic! Gonna take it for a spin tomorrow and I'll let you all know the results!

Antidote

Edit history:

Antidote: 2016-02-21 04:52:42 am

Antidote: 2016-02-21 12:22:57 am

2016-02-21 12:21:58 am

*innocently browsing the rumble code*
*looks in CRumbleManager::StopRumble*

.......

WHY is it calling beqlr THREE TIMES IN A ROW??????

Aruki

2016-02-21 01:47:49 am

Not really new info, but for reference here's the list of file formats from the MP3 prototype. It contains both Retro's internal uncooked formats as well as the cooked formats actually in the games.

Code:

mb - Maya native binary file
tga:jpg:jpeg - Raw texture file
rtr - Intermediate format data
artr - Raw animation data
prtr - Raw portal data
base - Base character pose
sml - Character Segment Master List
bptk - Character body parts 'to keep' list
crtr - Raw collision data
clsn - Cooked collision data
lctr - Locator data
cspp - Cooked spatial primitive
cmdl - CModel instance
cskr - Character skinning rules
txtr - Texture MipMap data
ani - Character animation data
cin - Character Information data
PLTT - RGraphics color palette
rpff - RFont raster font
acs - Animation Character Set
anim_set - Raw Animation Set data
ced - Character Editor file
mad - Meta-animation Data File
we2:wld - World Editor 2 Data File
mwld - Metroid World
mrea - Metroid Area
mapw - Metroid World Map
mapa - Metroid Area Map
savw - Metroid World Save Info
sava - Metroid Area Save Info
part - Particle System
wpsc - Weapon Particle System
swhc - Swoosh Particle System
dpsc - Decal Particle System
elsc - Electric Particle System
crsc - Collision Response Particle System
spsc - Spawn Particle System
srsc - Sorted Particle System
pas_def:pdef - Parameterized Animation State Data
fsm - Ai Finite State Machine Raw
afsm - Ai Finite State Machine
fsm2 - New Finite State Machine
dcln - Dynamic Collision Data
agsc - Cooked Audio Group Set
agsr - Raw Audio Group Set
refLight - Reference Light
sl - Scripting Layer
vis - Potential Visible Set
pfl2 - Pathfinding layer 2
path - Pathfinding
ptla - Portal Area
dgrp - Asset Dependency List
hght - Heightmap
drob - RGL compatible static-geometry object
skob - RGL compatible skinned object
rmtt - RGL compatible material settings
rptg - RGL compatible SceneGraph portal data
shader - RGL compatible shader resource
rtex - RGL compatible texture resource
atbl - Audio Lookup Table
song - MusyX Midi file
csng - Cooked MusyX Midi File
strg - Stringtable Asset
scan - Scannable Asset
frme - Gui Frame
stlr - Raw String List
stlc - Cooked String List
egmr - Raw EditorObject to StaticGeometry Map
egmc - Editor Object to Static Geometry Map
xml - Generic XML file
rule - Cooked Rule Asset
char - Cooked Character Data
sand - Cooked Source Animation Data
gml - Geometry Master List
wav:aif:aiff:mp3:ogg - Raw Audio File
csmp - Cooked Audio Data
aprj - Audio Manager Project
gob - Audio Manager macro
caud - Cooked Audio Manager macro

Not really new info, but for reference here's the list of file formats from the MP3 prototype. It contains both Retro's internal uncooked formats as well as the cooked formats actually in the games.

[code]mb - Maya native binary file
tga:jpg:jpeg - Raw texture file
rtr - Intermediate format data
artr - Raw animation data
prtr - Raw portal data
base - Base character pose
sml - Character Segment Master List
bptk - Character body parts 'to keep' list
crtr - Raw collision data
clsn - Cooked collision data
lctr - Locator data
cspp - Cooked spatial primitive
cmdl - CModel instance
cskr - Character skinning rules
txtr - Texture MipMap data
ani - Character animation data
cin - Character Information data
PLTT - RGraphics color palette
rpff - RFont raster font
acs - Animation Character Set
anim_set - Raw Animation Set data
ced - Character Editor file
mad - Meta-animation Data File
we2:wld - World Editor 2 Data File
mwld - Metroid World
mrea - Metroid Area
mapw - Metroid World Map
mapa - Metroid Area Map
savw - Metroid World Save Info
sava - Metroid Area Save Info
part - Particle System
wpsc - Weapon Particle System
swhc - Swoosh Particle System
dpsc - Decal Particle System
elsc - Electric Particle System
crsc - Collision Response Particle System
spsc - Spawn Particle System
srsc - Sorted Particle System
pas_def:pdef - Parameterized Animation State Data
fsm - Ai Finite State Machine Raw
afsm - Ai Finite State Machine
fsm2 - New Finite State Machine
dcln - Dynamic Collision Data
agsc - Cooked Audio Group Set
agsr - Raw Audio Group Set
refLight - Reference Light
sl - Scripting Layer
vis - Potential Visible Set
pfl2 - Pathfinding layer 2
path - Pathfinding
ptla - Portal Area
dgrp - Asset Dependency List
hght - Heightmap
drob - RGL compatible static-geometry object
skob - RGL compatible skinned object
rmtt - RGL compatible material settings
rptg - RGL compatible SceneGraph portal data
shader - RGL compatible shader resource
rtex - RGL compatible texture resource
atbl - Audio Lookup Table
song - MusyX Midi file
csng - Cooked MusyX Midi File
strg - Stringtable Asset
scan - Scannable Asset
frme - Gui Frame
stlr - Raw String List
stlc - Cooked String List
egmr - Raw EditorObject to StaticGeometry Map
egmc - Editor Object to Static Geometry Map
xml - Generic XML file
rule - Cooked Rule Asset
char - Cooked Character Data
sand - Cooked Source Animation Data
gml - Geometry Master List
wav:aif:aiff:mp3:ogg - Raw Audio File
csmp - Cooked Audio Data
aprj - Audio Manager Project
gob - Audio Manager macro
caud - Cooked Audio Manager macro[/code]

DJGrenola

2016-02-21 04:39:06 am

Quote from Antidote:

WHY is it calling beqlr TREE TIMES IN A ROW??????

that's an odd one. see if the same sequence occurs anywhere else?

Antidote

Edit history:

Antidote: 2016-02-21 04:53:29 am

2016-02-21 04:52:02 am

Not that I've seen, I think it was CW derping pretty badly.

EDIT:
Running it in a debugger doesn't reveal anything, if it passes the first one, it passes all of them.

DJGrenola

2016-02-21 05:02:35 am

only thing I can think of is that it was done to force the alignment of something lower down in the function. but I don't know why that would be.

Aruki

2016-02-21 05:03:37 am

Maybe the original code had some redundant checks that CodeWarrior optimized out, but it left in the beqlr for some reason?

DJGrenola

2016-02-21 05:18:48 am

it's as plausible as any other explanation I suppose. it's just a bit intriguing because if it were a compiler bug, you'd expect to see similar code in other places. I'm wondering if that's definitely retro's code and not something that was provided as part of the GC dev kit (which could have been compiled with something other than CW), but I'm guessing -- you folks would have a better idea about that than I would.

Aruki

2016-02-21 05:55:48 am

Well, it's Retro's class and Retro's name conventions, so it's probably their code unless there's an inlined call to an SDK function somewhere right at the beginning. Which could potentially be the case if that function interacts directly with hardware (to stop controller rumbling), but Antidote would know more about the actual contents of that function than me.

Antidote

Edit history:

Antidote: 2016-02-21 11:51:51 am

2016-02-21 11:51:35 am

No inlining unfortunately, this is either a one off compiler bug (unlikely), Retro screwed up (slightly more likely), or CPU behavior that's undocumented (entirely possible).

jackoalan

2016-02-21 02:22:21 pm

Intriguing.. here's an experiment compiled with 2001-era CodeWarrior for Macintosh PPC target (Optimization Level 4)

Looks like Retro wanted to be ABSOLUTELY CERTAIN that id isn't -1.. triple-checking for such.

wowsers

Edit history:

wowsers: 2016-02-21 02:41:34 pm

wowsers: 2016-02-21 02:39:35 pm

2016-02-21 02:39:06 pm

The only thing I could think of is that it's made to delay the "clear left immediate" for some obscure threading bug or similar, to buffer cpu cycles some.
Would be interesting to see where/what/how it would effect things.

jackoalan

2016-02-21 02:47:19 pm

PPC registers are much more stateful than that, it should be one-and-done.. Not to mention MP is strictly a single-threaded game (all interrupt-driven).

It seems more like a preprocessor faux-pas to me, perhaps a recursive controller-id filter using #define. The GCN SDK isn't involved at least until the Update method called outside this function.

wowsers

2016-02-21 02:55:56 pm

That could very well be it, I'm so glad we have async functions today so we don't have to deal with single thread horrors (at least in most languages). If I wasn't busy with school/game development and I had the knowledge I would totally check out what would happen if one used a single if statement.

DJGrenola

2016-02-21 04:27:04 pm

very nice.

does indeed look like some #ifdef fail.

Antidote

Edit history:

Antidote: 2016-02-21 04:38:39 pm

2016-02-21 04:34:15 pm

XD that's hilarious

Retro be like

Aruki

2016-02-22 10:38:01 am

I'm able to edit and resave levels from the entire Prime series now. Couple quick tests from MP2/3:

My first attempt at resaving DKCR didn't go so well, unfortunately. It didn't crash, but the camera's at 0,0,0 and the game is softlocked. Nothing happening except ambient background noises. Not sure what's going on but hopefully I can figure it out. Getting the default property values was a huge pain so I really hope I wasn't on the wrong track, or at least that there's not too much more to it than that.

Aruki

2016-02-23 12:53:57 pm

I managed to get DKCR working!

The editor doesn't track any info as far as which properties were originally in the file, so this is just going off the default property values I extracted before, and for the most part the level plays exactly like it's supposed to. The only problem I've noticed so far is that CameraHints are broken, which makes the level unplayable about halfway through (there's a CameraHint that's supposed to keep the camera in a fixed location and it doesn't deactivate when it's supposed to), so I need to get that sorted out. But aside from that, DKCR resaving works!

Antidote

2016-02-23 03:47:26 pm

NICE!, sucks about the camera hints though :/

Aruki

2016-02-24 12:37:03 am

The camera issues are fixed now, everything seems to work as it should.

jackoalan

2016-02-28 10:01:55 pm

OK, after months of neglecting the wiki, here's a major addition: Particle Element Docs!!
http://www.metroid2002.com/retromodding/wiki/Particle_Script

Used to be a bunch of guesses based on tweaking values and observing changes, now it's all properly reverse-engineered discoveries.

Antidote

2016-02-28 10:03:54 pm

Still, for being a bunch of guesses, you were pretty darn close, a few errors like "SPOS" being totally wrong, but not bad for guesses.

DJGrenola

2016-03-01 05:51:22 am

so I spent a few days looking at no-effort font replacement -- specifically, something that will take a .FONT file from the game and a TrueType or OpenType font file as input, and spit out a new .FONT file plus a .TXTR to be repacked into the game.

it isn't perfect yet, but it works.

attachment:

screenshot.png