Metroid Prime hacking research

2015-02-10 05:23:46 pm

You say that after a submesh header, the primitive data starts. Then a flag byte and after that a 16-bit value which indicates the vertex count. But this number is always 6 or F, how could that ever be a vertex count? I am actually quite noob when it comes to 3d models.

Aruki

2015-02-10 05:24:30 pm

What file/offset are you looking at?

Jesse

2015-02-10 05:25:27 pm

A CMDL file, at the submesh section

Aruki

2015-02-10 05:25:52 pm

Yeah, but which CMDL file at specifically which offset? I wanna pull it up for myself and see what you're talking about.

Jesse

Edit history:

Jesse: 2015-02-10 05:32:55 pm

Jesse: 2015-02-10 05:32:39 pm

2015-02-10 05:30:53 pm

Oh sorry xD

edit: Sorry I meant
> L02_Jungle_Cling.pak
> 16E82709A705EE21.CMDL
> offset 0x541

Aruki

2015-02-10 05:43:36 pm

Ok, so yeah, that's the vertex count - 0xF. That'll be followed by 0xF vertices, then another primitive will start. You'll want to keep reading it til you either hit the end of the section or read a flag value of 0. You'll notice the material ID for this submesh, at 0x53A, is 0 (which makes sense, because there's only one material). That material has an attrib flag setting of 0x30F. Each pair of bits on that value indicates a vertex attribute. 0x30F indicates position, normal, and one UV coord; so three attribs per vertex. So following that 0xF, we'll have 2 bytes per attribute, for 6 bytes per vertex; with 15 vertices, that means 90 bytes for this first primitive. So this primitive is 0x5A bytes long; it ends at 0x59D, at which point you'll find another flag value/vertex count marking the start of the next primitive. Does that make sense?

Jesse

2015-02-10 05:56:35 pm

This makes sense yeah, Thanks for explaining. I couldn't have found it out on my own.

RetroLover

Edit history:

RetroLover: 2015-02-11 12:14:19 am

2015-02-11 12:06:10 am

Quote from Paraxade:

If you worked on Tropical Freeze could you maybe tell us why there are 128-bit file IDs in that game? Because just.. seriously.. why.

All I can do is pray that the next Retro game won't use 256-bit IDs.

Sure thing. We rewrote a lot of stuff for TF, one of those was our resource management system. We changed it so references to resources were by GUID instead of filepath. This allowed us to move resources around the file system, without having to fix up all the references to it (which would have been the case if they were referenced by path). Instead we just had to fix up the GUID->File path lookup in the resource database. Because all resources had a GUID and we just utilized that for referencing resources in the runtime as well, thus the 128-bit IDs. Don't worry, it will most likely never increase.

RetroLover

2015-02-11 12:07:07 am

Also good luck with figuring out ANIM. That engineer does some interesting stuff.

Aruki

2015-02-11 12:10:41 am

Oooh I see, that makes sense. Thanks for the explanation!

Lord Fredrik

2015-02-11 12:09:53 pm

RetroLover works for Retro? :O

Anyway, I just found out I can't texture worth crap. Hopefully, sometime in the future, the textures will be rippable.

Jesse

2015-02-11 01:01:25 pm

You can help him reach that future.

Aruki

2015-02-11 01:05:03 pm

I haven't researched it too thoroughly but from what I've seen they've made the swizzling a lot more complicated in GX2. The techniques that were used with GX won't work; it'll take a lot more to get textures out of there. :/

For reference here's what a Tropical Freeze texture looks like without any unswizzling whatsoever.

5004e7afc92042e5a247a014aecc70d7.TXTR

Lord Fredrik

2015-02-11 01:28:52 pm

Quote from Jesse:

You can help him reach that future.

I would gladly help him, Jesse. Except I know fuck all about Retro game hacking. I've spent 7 years hacking Smash Brawl, it's all I know.

All I'm interested in is ripping assets lol.

And that looks pretty crazy, Parax

Antidote

2015-02-11 05:01:15 pm

It looks like it's either a 2x4 or 4x2 swizzle pattern. If so those are some tiny blocks.

SkippyJr

2015-02-12 02:10:41 pm

The start of ANIMs can be parsed by my load-the-game-as-a-library tool. But it crashes after a repeated structure even though its in a previously used codepath. Preliminary stuff can be posted if anyone wants.

Quote from Paraxade:

edit 4: There was another piece missing of the third table. In the long/float/float/long, the first long is another count value which is followed by a number of 12-byte-long structures. That count/table is in every file, while the second one (after the floats) is only in 3/4. The ANIM/EVNT pairs are only in 4. I've been able to parse the whole file now, and I'm running it on other files as well with no issues so far.

Do you think you could point me to where this code is in the demo disassembly, Skippy? I've got most of the functions labeled so I might be able to find what some of this data actually is. Also, how many unknown struct types are there, if you know?

If you (or anyone else) wants I can report what happens in a given byte range in whatever files you want via PM/email/post. Going back to the same code in the demo would be easy enough, just tell me a file that does the interesting thing you are looking at. In the tail end of ANCS which has been looked at recently, there are two types of structs and the switch(type) for both looks like the numbers could go 0-5. (It explicitly looks for 1-4 and <1, >4 via a binary search version of switch().)

So DKCR and apparently Sonic 3D World are unstripped? If there are no tools which can currently read the debug symbols I can look into modifing doltool to create ELF files with the needed stuff.

The start of ANIMs can be parsed by my load-the-game-as-a-library tool.  But it crashes after a repeated structure even though its in a previously used codepath.  Preliminary stuff can be posted if anyone wants.

[quote=Paraxade]edit 4: There was another piece missing of the third table. In the long/float/float/long, the first long is another count value which is followed by a number of 12-byte-long structures. That count/table is in every file, while the second one (after the floats) is only in 3/4. The ANIM/EVNT pairs are only in 4. I've been able to parse the whole file now, and I'm running it on other files as well with no issues so far.

Do you think you could point me to where this code is in the demo disassembly, Skippy? I've got most of the functions labeled so I might be able to find what some of this data actually is. Also, how many unknown struct types are there, if you know?[/quote]
If you (or anyone else) wants I can report what happens in a given byte range in whatever files you want via PM/email/post.  Going back to the same code in the demo would be easy enough, just tell me a file that does the interesting thing you are looking at.  In the tail end of ANCS which has been looked at recently, there are two types of structs and the switch(type) for both looks like the numbers could go 0-5.  (It explicitly looks for 1-4 and <1, >4 via a binary search version of switch().)

So DKCR and apparently Sonic 3D World are unstripped?  If there are no tools which can currently read the debug symbols I can look into modifing doltool to create ELF files with the needed stuff.

Aruki

2015-02-12 02:16:32 pm

Out of the games I've seen, Sonic Lost World, Sonic Boom, Tropical Freeze, and Hyrule Warriors are all unstripped. Most Nintendo-developed games (3D World, Smash 4, Pikmin 3, etc.) aren't. Unstripped games seem to be common on the Wii U for some reason. There's some existing tools to convert RPX files to elf (which IDA can disassemble), but none of them work 100% properly that I've tried; one of them doesn't work at all in IDA and one doesn't maintain the symbol data properly. If you could get it working that'd be fantastic. I'll shoot you a PM.

SkippyJr

2015-02-12 06:12:27 pm

Dang you are fast. And shorter now. The next game of a certain developer is likely to be stripped now, but much like the Prime Demo we'll get good insight from Tropical Freeze on file formats and stuff.

Found only one tool, and it has source: https://github.com/Hykem/rpl2elf
The first pass at playing with that shows that there is much to go. I'll mess with it to see what can be done.

BTW apparently an IDA pluggin is being worked on:
https://gbatemp.net/threads/wii-u-hacking-discussion.367489/page-143#post-5302200

Aruki

Edit history:

Aruki: 2015-02-12 06:37:47 pm

Aruki: 2015-02-12 06:36:48 pm

2015-02-12 06:19:24 pm

Maybe, but hey, maybe they'll be nice to us...! Also I tend to leave the thread open in a tab so I usually see the title bar change when someone posts. And yeah I asked ark to shorten my name earlier :P

Didn't know there was an IDA plugin in progress, neat. That'll definitely make things easier later down the road.

edit: On an unrelated note, working a little more on repacking. So I've known for a while that only certain formats are compressed within paks, but I noticed a while ago that when it comes to particles, some are compressed and some aren't. Never looked into why. Today I worked it out - particles (PART, ELSC, SWHC, WPSC, DPSC, CRSC) are only compressed when their uncompressed size >= 0x400. Problem is, we can't replicate this 100% with extracted files because any particle ranging from 0x3E1 to 0x3FF will be padded to 0x400. In the original paks, these would be uncompressed, but in the repacked ones, it's not very easy to tell the difference between them (which shouldn't be compressed) and a particle that's exactly 0x400 bytes long (which should be).

However I went through and manually deleted the padding on all particles falling within that range, and ended up generating a pak that was byte-for-byte identical to the original. Yay accuracy!

On a side note all those formats are supposed to end with "_END", so it'd theoretically be safe to cut out all 0xFFs at the end of the file to get the correct unpadded size. I'm not sure it's really worth the effort though.

Maybe, but hey, maybe they'll be nice to us...! Also I tend to leave the thread open in a tab so I usually see the title bar change when someone posts. And yeah I asked ark to shorten my name earlier :P

Didn't know there was an IDA plugin in progress, neat. That'll definitely make things easier later down the road.

edit: On an unrelated note, working a little more on repacking. So I've known for a while that only certain formats are compressed within paks, but I noticed a while ago that when it comes to particles, some are compressed and some aren't. Never looked into why. Today I worked it out - particles (PART, ELSC, SWHC, WPSC, DPSC, CRSC) are only compressed when their uncompressed size >= 0x400. Problem is, we can't replicate this 100% with extracted files because any particle ranging from 0x3E1 to 0x3FF will be padded to 0x400. In the original paks, these would be uncompressed, but in the repacked ones, it's not very easy to tell the difference between them (which shouldn't be compressed) and a particle that's exactly 0x400 bytes long (which should be).

However I went through and manually deleted the padding on all particles falling within that range, and ended up generating a pak that was byte-for-byte identical to the original. Yay accuracy!

On a side note all those formats are supposed to end with "_END", so it'd theoretically be safe to cut out all 0xFFs at the end of the file to get the correct unpadded size. I'm not sure it's really worth the effort though.

Antidote

Edit history:

Antidote: 2015-02-12 11:53:36 pm

2015-02-12 11:48:47 pm

Not long ago I wrote a simple utility to extract the sound macros from AGSC files, and MrSinistar has found it exceedingly useful, so here it is:
http://dl.dropboxusercontent.com/u/21757902/mxmextract.zip

This is currently only a windows binary release, the source needs to be cleaned up before I release it. It doesn't have a license file in it but, it's being released under GPLv3.

Edit: Two further things: it only supports MP1, and to use it just drag the AGSC on to it, and it'll create a couple of directories, the first one should be Audio, the second is whatever the "module name" is e.g: Sheegoth.

Baby Sheegoth

Edit history:

Baby Sheegoth: 2015-02-13 10:51:52 am

2015-02-13 10:28:38 am

Holy hell, crazy that we get people from Retro posting here. That is wild. Is there anything interesting you can say about working on Metroid Prime?! :D

RetroLover

Edit history:

RetroLover: 2015-02-14 05:25:50 pm

2015-02-14 05:25:40 pm

It was fun. (Is that interesting?)

Baby Sheegoth

2015-02-15 09:12:00 pm

Quote from RetroLover:

It was fun. (Is that interesting?)

It is interesting (but not surprising!)

Jesse

2015-02-16 05:01:41 pm

So yeah I made a (bad copy) STRG editor for DKCR (And probably MP3 too), the other STRG editor didn't support this version did it? However, It doesn't support STRG with more fields yet. Because I cannot figure out how this table works, also every pointer gets calculated from a different point, instead from the beginning of the strings(Like with one field). Do you guys have an idea?

Antidote

Edit history:

Antidote: 2015-02-17 01:11:54 am

Antidote: 2015-02-17 01:05:36 am

Antidote: 2015-02-17 01:03:49 am

Antidote: 2015-02-17 01:03:38 am

2015-02-17 01:02:15 am

The table immediately following the language FOURCC list follows this format:

Code:

struct
{
    uint tableSize; // total size of strings
    uint stringOffsets[stringCount];
} lang[languageCount];

The offset is calculated from the beginning of the string pool, so in that example offset 0 points to "1-2 King of Cling".