Metroid Prime hacking research

Antidote

Edit history:

Antidote: 2015-02-09 12:05:13 pm

Antidote: 2015-02-09 12:04:53 pm

2015-02-09 12:04:24 pm

Quote from Paraxade:

DKCR uses the same STRG format as Prime 3, which yeah, there's no article on. I've never looked into that version of the STRG format myself. I think Antidote has but I'm not sure he's completely cracked it either.

Yeah, there is a set of tables right before the strings, and it's currently unknown how to associate strings with a particular language.

Jesse

2015-02-09 12:17:11 pm

Quote from Antidote:

Quote from Paraxade:

DKCR uses the same STRG format as Prime 3, which yeah, there's no article on. I've never looked into that version of the STRG format myself. I think Antidote has but I'm not sure he's completely cracked it either.

Yeah, there is a set of tables right before the strings, and it's currently unknown how to associate strings with a particular language.

Its in chronological order. If a langauge like dutch is not supported, then the pointer will point at the english version. If that is what you are refering to.

But I do not understand this thing. first you have normal STRG files.

But then you got this very large one, which has an extra row. or are these the "names"?

Aruki

Edit history:

Aruki: 2015-02-09 12:42:28 pm

Aruki: 2015-02-09 12:32:38 pm

Aruki: 2015-02-09 12:23:37 pm

2015-02-09 12:20:29 pm

Yeah, those are the names. It's probably similar to the ones from Echoes, which are documented on the wiki.

EDIT: You can see in that screenshot you posted - there's a name count at 0x10, which is 0 in the first picture and 9 in the second. Then the name table size at 0x14, which is 0 in the first picture and 0xBE in the second. Then it's followed by 9 u32 pairs - string offset (relative to start of table, after size value) and string index. Then 9 zero-terminated name strings, case-sensitively sorted in alphabetical order. It's exactly the same as Echoes.

Antidote

Edit history:

Antidote: 2015-02-09 01:20:09 pm

Antidote: 2015-02-09 01:16:37 pm

Antidote: 2015-02-09 01:15:53 pm

Antidote: 2015-02-09 12:51:05 pm

Antidote: 2015-02-09 12:50:35 pm

Antidote: 2015-02-09 12:38:20 pm

2015-02-09 12:35:53 pm

http://pastebin.com/0JV1B8DP <- got it figured out already.

XD, Wish I had seen your post jesse, but I did manage to figure it out.
The value you're pointing at in the second shot is the string count. The one immediately following it controls the string names.

EDIT:
Oh, the Italian translation got the pronoun wrong according to google translate (take that with a grain of salt), any Italian speakers that can verify?
https://translate.google.com/#auto/en/Sta%20rinvenendo.%20Samus%2C%20riesce%20a%20sentirmi%3F

Aruki

Edit history:

Aruki: 2015-02-09 01:33:07 pm

2015-02-09 01:32:38 pm

Well I'm trying to make small tweaks to the Jungle Intro MLVL just for the sake of trying to change something, but there's barely anything in there for me to change. Best thing I have managed to do is crash the game by changing the MREA ID. I was trying to change the level name but that doesn't work... come to find out it's controlled by a giant STRG file in FrontEnd.pak. So bleh.

It is still an important file just because it provides access to the MREA, but it seems like in DKCR it doesn't do much other than that.

Jesse

2015-02-09 01:56:33 pm

You can try to edit the STRG id or the SAWV id. What does SAWV actually do? it sounds like its a save file but that would be odd.

Aruki

Edit history:

Aruki: 2015-02-09 02:01:10 pm

2015-02-09 01:58:51 pm

SAVW is kinda neat actually. It's a system that allows the save file to track non-hardcoded data. In Metroid Prime you can use it to make the save file remember what things you've scanned, what type of data is in the logbook, what cutscenes are skippable, what memory relays have been toggled, etc. I'm not sure what it tracks in DKCR but I'd imagine it serves a similar purpose.

Also as far as I can tell the STRG ID doesn't do anything, because that STRG seems to never actually display ingame. It uses the one from FrontEnd.pak for that. The MLVL in DKCR seems to contain a lot of stuff that's just largely left over from Metroid Prime :/ Not surprising they ditched it in DKCTF.

Antidote

2015-02-09 03:07:15 pm

Yeah, SAVW is pretty ingenious, and I'm surprised that not many games use such a system. It makes the save system incredibly robust and extensible.

Antidote

Edit history:

Antidote: 2015-02-09 08:16:38 pm

2015-02-09 07:13:01 pm

Figured out what the 16 bytes after the first two unknowns in MP3/DKCR paks are, it's an MD5 hash of the data chunks including the chunk table. So ignore the header, then hash the rest.

Aruki

2015-02-09 11:46:31 pm

hey Jesse, I did half the CMDL article here. I'll do the rest tomorrow.

RetroLover

2015-02-09 11:58:22 pm

Just created an account to post! The reverse engineering of this is pretty amazing, especially the reverse engineering of some of the formats that I created at Retro such as ROOM for DKCTF. Anyways, I probably won't post much (if at all after this), but Retro is looking for some talented and passionate engineers. If you guys are willing to reverse engineer all of this, it might be more fun to make it from scratch there as it's a great place to work. Passion goes a LONG way at Retro, so give it a shot. If you mention that this is your hobby, you'll most likely get at least a phone call. Later! http://retrostudios.com/careers.asp

Aruki

2015-02-10 12:06:02 am

omg geez. Thanks for stopping by! That's awesome to hear.

For me personally it is not exactly much more than a hobby right now. I don't do this professionally and I'm not professionally trained in any of this so I think I would most likely be dead weight at Retro. Maybe in the future I'll pursue it more seriously and that'll change. But thank you for the kind words!

RetroLover

2015-02-10 12:11:40 am

So, here's the deal. If what I've read from your posts is your actual work, you would be far from a dead weight. While you might not be professionally trained, you show raw talent and passion and could be fairly easily be polished. The engineering manager was not professionally trained either, he learned most of his skills from hard work and determination, so he can spot talent if it is there. Don't sell yourself short.

Just my two cents. Have a nice day!

Antidote

2015-02-10 12:13:20 am

Oh it's his actual work alright, we throw notes back and forth and check each other for errors, it's extremely nice to be able to work with someone that has the same interest as I do. Sure we don't always agree but our passion is the same.

Aruki

2015-02-10 12:17:03 am

Wow, thanks. It is my work but it's not really where I'd like it to be and I've only gotten as far as I have with a lot of help from others (I have some pretty talented friends who've given me a lot of advice with a bunch of things). I still don't think I am quite up for working at a place like Retro but if I do ever end up thinking about doing this professionally I'll remember what you said. Thank you!

RetroLover

2015-02-10 12:22:20 am

Well think about it! That goes for a number of you. Antidote and Jesse digging around the STRG format brings back good memories of my localization tool for Prime 3. It's great how you guys are pointing out all the differences between DKCR and DKCTF, lots (and lots and lots) of stuff changed.

Antidote

2015-02-10 12:23:15 am

Yeah, it's fun poking about the binaries in IDA and seeing the code evolve.

Aruki

2015-02-10 12:24:36 am

Tons. It's a little off-putting with how much is different honestly, since so little of our work with the previous games carries over to TF, heh.

At least we have an unstripped binary with Tropical Freeze! That was helpful with figuring out how the compression in models and textures worked :P Do us a favor and leave some symbol data in the next game you guys make for us, haha.

Aruki

2015-02-10 01:05:28 am

If you worked on Tropical Freeze could you maybe tell us why there are 128-bit file IDs in that game? Because just.. seriously.. why.

All I can do is pray that the next Retro game won't use 256-bit IDs.

Jesse

2015-02-10 09:27:18 am

> 3 games later
> 2048-bits IDs
however, Making file systems like this sounds like a lot of fun to do. Especially something like 3d files.

Anyways, I looked today in ANIM files. It seems that at least the first section has values which are split up in 4 16-bit values. There must be values for at least Location, Rotation and skill. I noticed that the first value is most of the time 0x0000. But I think I should save that for later since for as far as I know there is not much information on bones and how they connect to vertices, well atleast not for MP3 and DKCR...

Antidote

2015-02-10 10:53:48 am

ANIM uses CToken almost exclusively, and AFAIK it works with 32bit values, though you may be right.

Aruki

2015-02-10 10:54:04 am

Quote from Jesse:

> 3 games later
> 2048-bits IDs

oh god Retro please no

Bones are in CINF, and skin bindings are in CSKR. They're associated with the models via ANCS files (Prime 1/2) or CHAR files (Prime 3/DKCR). I imagine the formats probably haven't changed too much at least in MP3, but I do recall hearing in an interview somewhere they had to change the animation system a lot for DKCR, so I dunno. It's probably at least worth a try to compare the formats and see if there are similar structures in them compared to the mp1 formats.

Aruki

2015-02-10 02:40:14 pm

Jesse, I've updated the DKCR CMDL article... should cover about everything I've worked out now.

Jesse

2015-02-10 04:18:22 pm

Awesome, One thing though. Is everything chronological? because the section count can be differen't, how would you know what section is what section?

However, I looked at the material section. This should be a very easy one.

A material set starts with a 32 bit flag, Which actually seems to count how many sets are are being counted as one. For example, the CMDL header says there is one material set, but the first material flag has 0x02, that means that for some reason the next "Material set" is being counted with that other set. This second set has also not a flag. It might sound a bit weird how I explain it so you should take a look at that. but normally though a set should start with a flag.

Then there are some padding and stuff, I could not find any texture counter in these sets. you got also this magic called TRAN which is obviously transparency... But I guess you already know that. After all that you got the TXTR id and 64-bit padding. The set ends with 00 00 00 FF and END .

I hope my quick analyse was usefull.

Awesome, One thing though. Is everything chronological? because the section count can be differen't, how would you know what section is what section?

However, I looked at the material section. This should be a very easy one.

A material set starts with a 32 bit flag, Which actually seems to count how many sets are are being counted as one. For example, the CMDL header says there is one material set, but the first material flag has 0x02, that means that for some reason the next "Material set" is being counted with that other set. This second set has also not a flag. It might sound a bit weird how I explain it so you should take a look at that. but normally though a set should start with a flag.

Then there are some padding and stuff, I could not find any texture counter in these sets. you got also this magic called TRAN which is obviously transparency... But I guess you already know that. After all that you got the TXTR id and 64-bit padding. The set ends with 00 00 00 FF and END .

I hope my quick analyse was usefull.

Aruki

2015-02-10 04:23:47 pm

Yeah, it's always in the same order. In a CMDL the section order will always go Materials -> Vertex Coords -> Normals -> Colors -> Float UVs -> Short UVs -> Submesh Defs -> Submeshes. So the section count should always be 7 + the submesh count. In MREA where there's multiple meshes in the file, the vertex coords section for the next mesh is the section immediately after the final submesh of the previous mesh.

The way I figured out the materials in MP1 was by modifying Dolphin to dump out shader code, then making tweaks to the materials and seeing how it changed the resulting shaders. Then I was able to figure out a little bit more by disassembling the game. You could take a similar approach with the MP3/DKCR materials if you want, since at least with the MP1 materials it was rather difficult to tell what was what just from analyzing the file data alone. Either way I'm intending to take a look at it and figure it out with a similar approach soonish once I have a repacker for the MP3 pak format.

Yeah, it's always in the same order. In a CMDL the section order will always go Materials -> Vertex Coords -> Normals -> Colors -> Float UVs -> Short UVs -> Submesh Defs -> Submeshes. So the section count should always be 7 + the submesh count. In MREA where there's multiple meshes in the file, the vertex coords section for the next mesh is the section immediately after the final submesh of the previous mesh.

The way I figured out the materials in MP1 was by modifying Dolphin to dump out shader code, then making tweaks to the materials and seeing how it changed the resulting shaders. Then I was able to figure out a little bit more by disassembling the game. You could take a similar approach with the MP3/DKCR materials if you want, since at least with the MP1 materials it was rather difficult to tell what was what just from analyzing the file data alone. Either way I'm intending to take a look at it and figure it out with a similar approach soonish once I have a repacker for the MP3 pak format.