Metroid Prime hacking research

2014-09-09 06:38:53 am

Wannabe Everything

Quote from Antidote:

With duplicated assets? About 9GB, with them removed about half that.
With all textures exported to png, that's about 800MB alone.

Wow, I had no idea Prime 3 was such a large game. I had thought it more along the lines of 4-6GB.

Antidote

2014-09-09 06:41:23 am

I was surprised myself, but keep in mind, that's with duplicated assets, I've seen a texture duplicated 9 times in one pak. They do this for speed, the less you have to have that laser moving the faster things load.

Aruki

Edit history:

Aruki: 2014-09-09 06:46:59 am

2014-09-09 06:41:34 am

Nah, this is definitely the place for it so long as it's related to modding the game. Ask away, and feel free to PM someone if you think you're getting off-topic.

If it helps, I'm pretty much a noob at programming; I've dabbled in it from time to time for years, but I never got past the basics and never tried to code anything seriously until just the past 5-6 months. I ended up picking up Maxscript, first just writing a couple scripts to make my life a bit easier with modding Sonic Generations, then scripts that actually imported/exported some of its proprietary formats. Started poking around with Metroid Prime soon after that, using the code from MpxViewer as a reference to learn the MREA geometry format, and made a Maxscript importer for that. Fairly soon after that, I decided I had to pick up C++ again, since I'd learned the pak formt and decided I wanted an improved pak dumper with decompression functionality, and I couldn't exactly use zlib in Maxscript.

I focused on learning the basics of how the language worked at first, and when I felt I had a decent grasp on that (which didn't take long... probably due at least in part to prior experience), I started working on my unpacker, which eventually became PakTool. I'm honestly kind of surprised how much success I've had since then with more and more complex things, given my inexperience. I think having specific things you want to accomplish goes a long way in helping you maintain interest in learning and working on these things. It's also important IMO not to get too mired in absolutely mastering the language before trying to do anything with it; I think that's the reason why I never stuck with programming before. I could have tried to get really good at C++ before I tried coding anything serious with it, and I'd probably be writing better code if I had decided to do that and stuck with it, but I think it's a lot more likely I would have just lost interest and then not done anything at all.

So just start off learning the basics, try experimenting and focusing on the areas you're interested in once you feel comfortable with that, and let yourself learn on the way. That's pretty much my advice on starting out. Sorry for the wall of text, hopefully it was actually helpful :P

Nah, this is definitely the place for it so long as it's related to modding the game. Ask away, and feel free to PM someone if you think you're getting off-topic.

If it helps, I'm pretty much a noob at programming; I've dabbled in it from time to time for years, but I never got past the basics and never tried to code anything seriously until just the past 5-6 months. I ended up picking up Maxscript, first just writing a couple scripts to make my life a bit easier with modding Sonic Generations, then scripts that actually imported/exported some of its proprietary formats. Started poking around with Metroid Prime soon after that, using the code from MpxViewer as a reference to learn the MREA geometry format, and made a Maxscript importer for that. Fairly soon after that, I decided I had to pick up C++ again, since I'd learned the pak formt and decided I wanted an improved pak dumper with decompression functionality, and I couldn't exactly use zlib in Maxscript.

I focused on learning the basics of how the language worked at first, and when I felt I had a decent grasp on that (which didn't take long... probably due at least in part to prior experience), I started working on my unpacker, which eventually became PakTool. I'm honestly kind of surprised how much success I've had since then with more and more complex things, given my inexperience. I think having specific things you want to accomplish goes a long way in helping you maintain interest in learning and working on these things. It's also important IMO not to get too mired in absolutely mastering the language before trying to do anything with it; I think that's the reason why I never stuck with programming before. I could have tried to get really good at C++ before I tried coding anything serious with it, and I'd probably be writing better code if I had decided to do that and stuck with it, but I think it's a lot more likely I would have just lost interest and then not done anything at all.

So just start off learning the basics, try experimenting and focusing on the areas you're interested in once you feel comfortable with that, and let yourself learn on the way. That's pretty much my advice on starting out. Sorry for the wall of text, hopefully it was actually helpful :P

Antidote

Edit history:

Antidote: 2014-09-09 06:45:43 am

2014-09-09 06:43:11 am

My only beef with your code is that you use snake case ;P

Aruki

2014-09-09 06:46:20 am

I'm kind of inconsistent with that, I tend to use both snake case and camel case. Not sure if that's really a big deal though. snake is kinda easier to read to me.

DJGrenola

2014-09-09 06:48:14 am

no wonder prime 3 loads so slowly

also cool people use snake

Antidote

2014-09-09 06:49:13 am

weird people code like interdpth, it took me AGES to reimplement his code, and I've pretty much got none of it left now lol.

Aruki

2014-09-09 07:35:53 am

Okay, let's try this again. :P New version of PakTool. Aside from fixing the bug I mentioned before, where a lot of compressed Prime 3 textures were being dumped incorrectly, I've also made two functionality changes. The first is that a list file is now required in repack mode (and the syntax for this has slightly changed - the "-l" isn't needed anymore). Since MLVLs aren't being automatically stuck in the name table anymore, that means paks generated without a list file are not only inoptimal, but they won't even work... so there's really no sense in keeping that functionality around. The second change is just a minor one that will hopefully make things a little more convenient - dump mode will now reuse the name of the pak for the output text file, if no output name is explicitly set. The syntax is the same, but the "output file" argument is optional now.

Download attached to this post, and will also go in the first post momentarily.

I've tested this a fair bit and I'm pretty sure everything works properly, but lemme know if there are any problems.

Okay, let's try this again. :P New version of PakTool. Aside from fixing the bug I mentioned before, where a lot of compressed Prime 3 textures were being dumped incorrectly, I've also made two functionality changes. The first is that a list file is now required in repack mode (and the syntax for this has slightly changed - the "-l" isn't needed anymore). Since MLVLs aren't being automatically stuck in the name table anymore, that means paks generated without a list file are not only inoptimal, but they won't even work... so there's really no sense in keeping that functionality around. The second change is just a minor one that will hopefully make things a little more convenient - dump mode will now reuse the name of the pak for the output text file, if no output name is explicitly set. The syntax is the same, but the "output file" argument is optional now.

Download attached to this post, and will also go in the first post momentarily.

I've tested this a fair bit and I'm pretty sure everything works properly, but lemme know if there are any problems.

DJGrenola

2014-09-09 04:20:09 pm

parax do you happen to have a list of known PAK version numbers? specifically I'm wondering if the leaked versions of prime 1 and 2 still use version 0x00030005.

Aruki

2014-09-09 04:21:24 pm

uhh, you mean the demos? Those use the same pak formats as the final game, yeah

DJGrenola

2014-09-09 04:21:47 pm

yeah, ok, thanks

Antidote

2014-09-09 04:51:39 pm

There are only 2 pak versions 0x00030005 and 0x00000002.
Unfortunately the leaked beta uses ^ with 64bit resourceIDs, so supporting those paks is a bit of a hack.

DJGrenola

2014-09-09 04:54:41 pm

so no sane version numbering? you'd think there'd be a change for LZO paks and then another one for the 64 bit ID ones, jeez!

Aruki

2014-09-09 04:56:01 pm

are we sure they're even version numbers at all? Could just be some other constant value, which would explain why they didn't bother updating it

MilesSMB

2014-09-09 04:56:03 pm

What kind of version number is 0x00030005 anyway?

Aruki

2014-09-09 04:56:32 pm

Miles, get back to work on Echoes Randomizer v0x00030006

DJGrenola

2014-09-09 04:56:53 pm

I'd just assumed it was 3.5, I've seen them split like that before, but you're probably right, it probably isn't that at all

Antidote

2014-09-09 05:28:32 pm

Who knows, all I know is that the engine checks that value specifically.

Aruki

2014-09-09 06:01:56 pm

Support for Donkey Kong Country Returns added, at Miles's request.

attachment:

PakTool.rar

MrSinistar

2014-09-09 11:38:18 pm

Found something interesting in AGSC. The first two bytes that are not the audio file name or zeros seem to point very close to the end of the first chunk. For example, the Metroid audio file, 52ab7324.AGSC has it's first two bytes of data as 11 54. When you go to 0x1154, the next row has the end for the chunk. I tested this out with three other AGSCs and the end of the chunk (FF FF FF FF) is either on the same row, the next row or the next, next row. Granted it's not too accurate but still very close by.

Aruki

Edit history:

Aruki: 2014-09-09 11:42:08 pm

2014-09-09 11:41:33 pm

Yeah, we established that a while ago... it's the size value for the first chunk. Every chunk starts with a size value.

It's a size value, not an absolute offset, so you need to go 0x1154 bytes forward from the end of the size value.

MrSinistar

2014-09-09 11:43:46 pm

Ahh, so that's what you guys found out a few pages earlier. Sorry. We need to have a central source of documentation or something, lol.

Aruki

2014-09-09 11:51:33 pm

Unrelatedly, I went poking through a couple of the demo's files and found what seems to be another unused morph ball.

MrSinistar

2014-09-09 11:55:52 pm

Awesome!! God, I love unused stuff :D

MrSinistar

Edit history:

MrSinistar: 2014-09-10 01:08:46 am

2014-09-10 01:06:16 am

Speaking of unused stuff, the MP3 E3 demo has the Annihilator beam sounds as CSMPs (as placeholders for the Nova Beam sound effects). And after looking at the Annihilator Beam's charge sound effect, I noticed that the loop ends before the end of the sound, effectively making the rest of the sound unused in the game. Basically it's just another iteration of the loop sound, but just slightly different since the original recording actually continued on after the loop, they just forgot to trim it.