Metroid Prime hacking research

Aruki

Edit history:

Aruki: 2015-02-18 12:09:25 pm

Aruki: 2015-02-17 10:11:39 pm

Aruki: 2015-02-17 10:10:59 pm

2015-02-17 10:10:27 pm

Ok, so trying to figure out the specifics of the Prime 3 compression setup in paks. This is proving to be pretty messy. :/

So in MP3 compressed data can be stored in multiple blocks. There can be 1-3 blocks used, and generally everything is one block except textures. With textures, they generally compress each part of the texture separately: the header, the palettes (for C4/C8), and the actual image data. I'm not sure why, but that's what they do. Problem is, some textures actually only use one block, and I'm having trouble figuring out the reason why. It's not based on the texture format, and it doesn't seem to be based on the size of the file. I'll try a little longer to figure it out, but if that fails, the game probably won't care if all textures are in one block. (It should at least work though it might be less efficient - hard to say without knowing why Retro did it this way)

There's also a one-byte value taking up the top byte of some of the compressed sizes. I'm not really sure what the point of this byte is but it seems to be consistent, with only four possible values:

- 1 block: block 1 has 0xA0
- 2-3 blocks: blocks 2/3 have 0xC0 if compressed, 0x40 if uncompressed
- byte is 0 in all other circumstances

By the way thanks Grenola for getting me to implement the ability to automatically unpack every pak from a given game. It's making it really easy to run tests across every pak in the game :P

With PakTool the state I'm currently at is - every pak format has full extraction/decompression support, MP1/2/3 proto are fully supported, 3/DKCR have a repacker that needs some debugging. DKCTF isn't going to have a repacker right now because there's no way to actually make use of it. I'm hoping to get this done soon. I want to try figuring out MP3 materials once the repacker is working fully.

Also this is the 1000th post in the thread, yay. (Though if you go by the permalink URLs, RetroLover's post up there was actually the 1000th one :P)

Ok, so trying to figure out the specifics of the Prime 3 compression setup in paks. This is proving to be pretty messy. :/

So in MP3 compressed data can be stored in multiple blocks. There can be 1-3 blocks used, and generally everything is one block except textures. With textures, they generally compress each part of the texture separately: the header, the palettes (for C4/C8), and the actual image data. I'm not sure why, but that's what they do. Problem is, some textures actually only use one block, and I'm having trouble figuring out the reason why. It's not based on the texture format, and it doesn't seem to be based on the size of the file. I'll try a little longer to figure it out, but if that fails, the game probably won't care if all textures are in one block. (It should at least work though it might be less efficient - hard to say without knowing why Retro did it this way)

There's also a one-byte value taking up the top byte of some of the compressed sizes. I'm not really sure what the point of this byte is but it seems to be consistent, with only four possible values:

- 1 block: block 1 has 0xA0
- 2-3 blocks: blocks 2/3 have 0xC0 if compressed, 0x40 if uncompressed
- byte is 0 in all other circumstances

By the way thanks Grenola for getting me to implement the ability to automatically unpack every pak from a given game. It's making it really easy to run tests across every pak in the game :P

With PakTool the state I'm currently at is - every pak format has full extraction/decompression support, MP1/2/3 proto are fully supported, 3/DKCR have a repacker that needs some debugging. DKCTF isn't going to have a repacker right now because there's no way to actually make use of it. I'm hoping to get this done soon. I want to try figuring out MP3 materials once the repacker is working fully.

Also this is the 1000th post in the thread, yay. (Though if you go by the permalink URLs, RetroLover's post up there was actually the 1000th one :P)

Aruki

Edit history:

Aruki: 2015-02-18 01:38:56 pm

2015-02-18 01:29:21 pm

Tried comparing compressed sizes with one block compared to two on textures. While there are some that save space by using two, most of them are bigger with two blocks... the ones that are smaller only save a few bytes and therefore are bigger anyway just by virtue of needing an extra 8 bytes of metadata per block. I'm really not understanding why this was done this way... it doesn't save space, it seems doubtful it allows the texture to be loaded faster since it's not done consistently, so what's the point?

It's also nothing to do with the dimensions of the image or the mipmap count, and like I said in the last post, it seems to have nothing to do with the size of the image data or the image format used in the texture.. so yeah, I can't really find any rhyme or reason to which textures use one block and which use two/three.

At this point I think I'll just be setting everything to use one block, unless we end up figuring out why textures are compressed this way and it's something that actually makes much of a difference. Will need to verify the game doesn't care but I don't see any reason why it would.

edit: Also I've verified that one-byte value I mentioned in the last post. The possible values I listed are 100% consistent and there aren't any other possible settings. So we sorta know what that is now at least.

Tried comparing compressed sizes with one block compared to two on textures. While there are some that save space by using two, most of them are bigger with two blocks... the ones that are smaller only save a few bytes and therefore are bigger anyway just by virtue of needing an extra 8 bytes of metadata per block. I'm really not understanding why this was done this way... it doesn't save space, it seems doubtful it allows the texture to be loaded faster since it's not done consistently, so what's the point?

It's also nothing to do with the dimensions of the image or the mipmap count, and like I said in the last post, it seems to have nothing to do with the size of the image data or the image format used in the texture.. so yeah, I can't really find any rhyme or reason to which textures use one block and which use two/three.

At this point I think I'll just be setting everything to use one block, unless we end up figuring out why textures are compressed this way and it's something that actually makes much of a difference. Will need to verify the game doesn't care but I don't see any reason why it would.

edit: Also I've verified that one-byte value I mentioned in the last post. The possible values I listed are 100% consistent and there aren't any other possible settings. So we sorta know what that is now at least.

Aruki

Edit history:

Aruki: 2015-02-19 02:27:50 am

Aruki: 2015-02-19 02:27:27 am

Aruki: 2015-02-19 02:21:24 am

Aruki: 2015-02-19 02:12:02 am

Aruki: 2015-02-19 02:10:20 am

Aruki: 2015-02-19 02:09:06 am

2015-02-18 05:39:08 pm

Okay, I've gotten custom paks 100% working in both MP3 and DKCR. Woo. I need to do some more testing but hopefully I can do a release soon.

Ran into a really strange issue in DKCR. I was testing with L01_Jungle_Intro.pak, so I unpacked the pak, dumped a file list, and then tried to repack it. Aaand... game crashed. Then I replaced the new pak with the original one. Still crashing. What. So it turns out that if there's a folder called "L01_Jungle_Intro" in the same folder as the pak, then it makes the game crash... not sure if I should make it output different folder names to get around that. Input welcome on that.

Also I did a dumb mod yesterday for fun since I realized it would be super easy to do, lol.

edit: more info on the DKCR thing. This seems to be a silly bug that Retro wouldn't have caught since they wouldn't have had folders like this lying around in the game's directory structure. In DKCR the world pak names are not hardcoded. There's a file called areas.lst that provides a list of world paks, with A. the MLVL ID, and B. the path to the pak. Example of one line from the file:

Code:

0x2D553D16CC07CD8E Worlds/W01_Jungle/L01_Jungle_Intro

note that this line does not end in ".pak". The game's simply being told to look for "L01_Jungle_Intro". If there's a folder there by that name, then it finds the folder instead of the pak, hence the crash. If you append ".pak" to the end of the line, then the pak loads correctly even with the folder there. Ideally if you're modding you would modify areas.lst to link directly to the pak so that bugs like this can't happen, but with PakTool being more general it can't really rely on the user having a modified areas.lst. The only way PakTool can really handle it is by naming the folders something different, by appending something to the end of the name. I'm not sure if it should do that by default or if the user should just be made aware of the problem in the readme or something so they can handle it themselves if the situation comes up.

[edit: Antidote's just suggested appending "_pak" or "-pak" to the end of the name and I like that suggestion so I'm gonna go with that.]

Okay, I've gotten custom paks 100% working in both MP3 and DKCR. Woo. I need to do some more testing but hopefully I can do a release soon.

Ran into a really strange issue in DKCR. I was testing with L01_Jungle_Intro.pak, so I unpacked the pak, dumped a file list, and then tried to repack it. Aaand... game crashed. Then I replaced the new pak with the original one. Still crashing. What. So it turns out that if there's a folder called "L01_Jungle_Intro" in the same folder as the pak, then it makes the game crash... not sure if I should make it output different folder names to get around that. Input welcome on that.

Also I did a dumb mod yesterday for fun since I realized it would be super easy to do, lol.

https://www.youtube.com/watch?v=FfRFKCxKbVo

edit: more info on the DKCR thing. This seems to be a silly bug that Retro wouldn't have caught since they wouldn't have had folders like this lying around in the game's directory structure. In DKCR the world pak names are not hardcoded. There's a file called areas.lst that provides a list of world paks, with A. the MLVL ID, and B. the path to the pak. Example of one line from the file:

[code]0x2D553D16CC07CD8E Worlds/W01_Jungle/L01_Jungle_Intro[/code]

note that this line does not end in ".pak". The game's simply being told to look for "L01_Jungle_Intro". If there's a folder there by that name, then it finds the folder instead of the pak, hence the crash. If you append ".pak" to the end of the line, then the pak loads correctly even with the folder there. Ideally if you're modding you would modify areas.lst to link directly to the pak so that bugs like this can't happen, but with PakTool being more general it can't really rely on the user having a modified areas.lst. The only way PakTool can really handle it is by naming the folders something different, by appending something to the end of the name. I'm not sure if it should do that by default or if the user should just be made aware of the problem in the readme or something so they can handle it themselves if the situation comes up.

[edit: Antidote's just suggested appending "_pak" or "-pak" to the end of the name and I like that suggestion so I'm gonna go with that.]

Jesse

2015-02-19 06:07:07 pm

I was wondering if you are actually focusing on also making an editor for dkcr or just something like that.

I want to make a more full editor for dkcr when I understand MREA. but that won't probably happen for a long time, since I only yet implanted a string editor which doesn't even save directly to the PAK (that means you have to import it later) due to a bug. I also tried to make a CMDL to obj converter but not every CMDL seems to follow the exact same rules, though your max script doesn't seem to have much if and else statments, yet it is able to import every model. So I guess I am far from finished xD

Aruki

Edit history:

Aruki: 2015-02-19 06:11:17 pm

2015-02-19 06:09:40 pm

I'm focused on making an editor for the Prime series mainly, would like to do DKCR too but it's probably better to limit my scope for right now.

The PakTool rewrite is just about finished, just testing to make sure everything works properly... I should be able to release tonight I think.

btw Jesse have you considered working with dumped paks instead? I kinda like that better personally; you can just dump out/decompress everything, edit whatever, and then generate a new pak from scratch using the files. That's what PakTool is built to do.

Jesse

2015-02-19 06:24:49 pm

Yeah I use your paktool every day. to find files and match them with other files and such. And yeah I think that if I would create a bigger editor I would use your technique or use your exe to dump everything first instead of reading everything from a pak directly. it would atleast take away problems I have now.

Aruki

Edit history:

Aruki: 2015-02-19 11:35:16 pm

2015-02-19 11:33:57 pm

Oh god I'm finally finished with this. Took way longer than I thought it would be everything finally works and I feel it's ready for a release. I also feel good enough about this release to call it 1.0, so woo for that!

> PakTool v1.0 download

Tons of changes in this version that should hopefully make working with paks easier. Here's the full changelog from the readme:

Quote:

v1.0 2/19/15
- Duplicate files are now ignored in unpack mode, significantly improving performance.
- File formats with embedded compression can now be decompressed in Extract Mode and recompressed in Repack Mode, allowing for the decompression of MREA files as well as Tropical Freeze models, textures, and shaders at unpack time.
- You can now place PakTool in a game's root directory and specify that game's title in Extract and List Dump modes to automatically unpack every one of that game's paks. (thanks DJGrenola)
- Repacking is now supported for Metroid Prime 3 and Donkey Kong Country Returns.
- Repacking now requires you to enter the game you're repacking for. This sets both the output pak format and the compression used, and replaces the old compression type parameter.
- A new "auto" mode has been added for Repack Mode; now you can just specify the pak name and PakTool will automatically set the files folder, the output pak name, and the list file. This only works if all those things use the default names, though!
- Tropical Freeze files with pak metadata now have it appended to the end of the file.

More detail on all this stuff is in the readme. Hope you guys find it useful!

Oh god I'm finally finished with this. Took way longer than I thought it would be everything finally works and I feel it's ready for a release. I also feel good enough about this release to call it 1.0, so woo for that!

[b]> [url=https://drive.google.com/file/d/0B9MLV21H7SDvcGc1QjRfSTFJNk0/view?usp=sharing]PakTool v1.0 download[/url][/b]

Tons of changes in this version that should hopefully make working with paks easier. Here's the full changelog from the readme:

[quote]v1.0 2/19/15
- Duplicate files are now ignored in unpack mode, significantly improving performance.
- File formats with embedded compression can now be decompressed in Extract Mode and recompressed in Repack Mode, allowing for the decompression of MREA files as well as Tropical Freeze models, textures, and shaders at unpack time.
- You can now place PakTool in a game's root directory and specify that game's title in Extract and List Dump modes to automatically unpack every one of that game's paks. (thanks DJGrenola)
- Repacking is now supported for Metroid Prime 3 and Donkey Kong Country Returns.
- Repacking now requires you to enter the game you're repacking for. This sets both the output pak format and the compression used, and replaces the old compression type parameter.
- A new "auto" mode has been added for Repack Mode; now you can just specify the pak name and PakTool will automatically set the files folder, the output pak name, and the list file. This only works if all those things use the default names, though!
- Tropical Freeze files with pak metadata now have it appended to the end of the file.[/quote]

More detail on all this stuff is in the readme. Hope you guys find it useful!

attachment:

PakTool.rar

Antidote

Edit history:

Antidote: 2015-02-19 11:37:10 pm

2015-02-19 11:36:46 pm

Confirmed working in Wine, before when I attempted to use the old version I would get 0 byte files for some weird reason.

jackoalan

Edit history:

jackoalan: 2015-02-20 08:56:04 pm

2015-02-20 07:52:08 pm

Hello fellow hunters!!
I'm new here, and have some contributions to make.

I've been focusing on reverse engineering the PART format and here are my findings so far:
http://www.metroid2002.com/retromodding/wiki/PART_(File_Format%29
http://www.metroid2002.com/retromodding/wiki/Effect_Script

I've even put together a viewer application to view standalone effects. It will eventually support the other effect types (SWHC, ELSC, DPSC).

Here is a rendering of the morph ball wall-spark effect (which runs at 60fps when not a GIF):
http://gfycat.com/ClassicSlimGoldenretriever

Antidote

2015-02-20 09:41:56 pm

Nice work, and thanks for the contributions, they're greatly appreciated.

Aruki

2015-02-20 09:47:47 pm

holy shit, haha. How long have you been working on this for? That's crazy. Really cool stuff.

jackoalan

2015-02-20 11:58:53 pm

I actually had no idea this community existed, and just discovered Parax's twitter last week and this thread + the modding wiki today.
That pushed me to publish my findings (which originally existed as Markdown docs, so I converted them to MediaWiki).

Quote:

holy shit, haha. How long have you been working on this for?

I started examining MP1's hex around mid-2013, using a Mac application I wrote called mPakView to access PAK contents.
http://imgur.com/PPo4nsS

I kept a handwritten journal of my general findings, which I refined into Markdown docs later.

Now for the bombshell:

In Dec-2013, I started putting together a codebase (written in C) of extractors for most of the data formats in the game (not just particles).
The extractors output data representations as Blender and Python files.

So far, It generates Blender files of:

* Areas (with almost every SCLY object represented in the scene and PATH navmeshes)
* Meshes
* Collision Meshes
* Map Meshes
* Actors (complete with CINF/CSKR rigging and ANIM animations)

and Python files of:

* Levels
* Area SCLY layers (paired with the area's .blend)
* Effect scripts
* Scan metadata
* Frame layouts
* Game strings (converted to UTF8)

Reverse engineering is much quicker when the data is in a nicer representation ;)

Here's a Blender screenshot of Artifact Temple with camera splines highlighted:
http://imgur.com/pT7a16g,6ShBhAu,T9zATGW

and collision geometry:
http://imgur.com/pT7a16g,6ShBhAu,T9zATGW#1

I'll post more updates, particularly when I'm ready to make the codebase public.

I actually had no idea this community existed, and just discovered Parax's twitter last week and this thread + the modding wiki today.
That pushed me to publish my findings (which originally existed as Markdown docs, so I converted them to MediaWiki).

[quote]holy shit, haha. How long have you been working on this for?[/quote]

I started examining MP1's hex around mid-2013, using a Mac application I wrote called [i]mPakView[/i] to access PAK contents. 
[url]http://imgur.com/PPo4nsS[/url]

I kept a handwritten journal of my general findings, which I refined into Markdown docs later.

[b]Now for the bombshell:[/b]

In Dec-2013, I started putting together a codebase (written in C) of extractors for most of the data formats in the game (not just particles).
The extractors output data representations as Blender and Python files.

So far, It generates Blender files of:

* Areas (with almost every SCLY object represented in the scene and PATH navmeshes)
* Meshes
* Collision Meshes
* Map Meshes
* Actors (complete with CINF/CSKR rigging and ANIM animations)

and Python files of:

* Levels
* Area SCLY layers (paired with the area's .blend)
* Effect scripts
* Scan metadata
* Frame layouts
* Game strings (converted to UTF8)

Reverse engineering is much quicker when the data is in a nicer representation ;)

Here's a Blender screenshot of Artifact Temple with camera splines highlighted:
[url]http://imgur.com/pT7a16g,6ShBhAu,T9zATGW[/url]

and collision geometry:
[url]http://imgur.com/pT7a16g,6ShBhAu,T9zATGW#1[/url]

I'll post more updates, particularly when I'm ready to make the codebase public.

Aruki

2015-02-21 12:02:52 am

Very nice. That's a really impressive amount of stuff to come up with by yourself. We don't have much of anything in PATH or ANIM worked out yet either, it'd be cool if you could share your findings on those as well!

jackoalan

2015-02-21 12:04:18 am

Certainly, it'll take some time to adequately explain ANIM, it's a tricky bastard aiwebs_007

Aruki

Edit history:

Aruki: 2015-02-21 12:15:20 am

2015-02-21 12:14:11 am

What led you to my twitter by the way? :P

Also what are you still missing info on? We have most of what we know on the wiki but there are some gaps (Wiki has only existed for a few weeks).

jackoalan

Edit history:

jackoalan: 2015-02-21 12:19:13 am

2015-02-21 12:18:10 am

I found some posts you made on metroid-contruction. One google search later, and there's your twitter up top.

As far as the missing info, there are lots of things I skipped over in the area SCLY section. I didn't see any docs on the wiki for those.
Also, I managed to get an AGSC audio player of my own working, but that first table with the articulator configs is only just started. The wiki is much more complete, so I'll refer to that.

Aruki

2015-02-21 12:20:09 am

SCLY's mostly figured out, it just needs some trial and error to figure out what the remaining object properties are. There's sorta some documentation on my sandbox page; we haven't gotten that onto the main wiki yet. http://www.metroid2002.com/retromodding/wiki/User:Parax0/Sandbox

AGSC's first table was cracked recently. http://www.metroid2002.com/retromodding/wiki/AGSC_(File_Format)#Pool

jackoalan

Edit history:

jackoalan: 2015-02-21 12:24:36 am

2015-02-21 12:22:06 am

Okay. I figured out almost all of DebrisExtended, so I'll add that to your Sandbox.

Also, I'd suggest adding a 'notes' column to the tables. Makes describing intricacies much easier.

Aruki

2015-02-21 12:29:28 am

Yeah that might be a good idea :P

Antidote

2015-02-21 12:43:54 am

Jeeze, just your rough notes on ANIM would make me a really happy man, also you figured out PATH? :O I've only gotten about the first half of it figured out.

jackoalan

Edit history:

jackoalan: 2015-02-21 01:04:53 am

2015-02-21 12:54:52 am

PATH is remarkably straightforward; It's an n-gon mesh.

Table 1 lists verts and normals
Table 2 lists polygon-edges with meta-data
Table 3 are the polygons themselves linking tables 1 and 2 and adding meta-data like polygon normals

I'll post the details for that as well

Here's what the PATH mesh ends up looking like for Main Quarry:
http://imgur.com/UZxzb1r

Antidote

Edit history:

Antidote: 2015-02-21 01:08:12 am

2015-02-21 01:04:57 am

I didn't think about n-gons I probably should have. Nice, that's pretty cool.

Aruki

2015-02-21 01:22:21 am

Have you done much with MP2/3 or have you mainly just focused on MP1?

jackoalan

2015-02-21 01:50:07 am

Just MP1 so far (using GM8E-0-00 and the NTSC Trilogy images)

Antidote

2015-02-21 02:44:20 am

I've made a decision that RetroView will permanently live by it's name and remain a viewer, PWE is getting extremely good and Parax has put a lot of effort into it, and having a competing piece of software in the same thread is pretty silly. So in the future it'll probably end up being like this: RetroView for people who simply want to view the assets and PWE for those who want to modify map layers.