Metroid Prime hacking research

Aruki

Edit history:

Aruki: 2017-02-07 06:58:08 pm

Aruki: 2016-03-24 04:36:53 am

Aruki: 2016-03-24 04:36:06 am

Aruki: 2016-03-24 04:35:57 am

Aruki: 2016-03-24 04:35:19 am

Aruki: 2016-03-24 04:33:42 am

Aruki: 2016-03-24 04:33:23 am

Aruki: 2016-03-24 04:33:11 am

Aruki: 2016-03-22 10:43:39 pm

Aruki: 2016-02-16 05:31:27 am

Aruki: 2016-01-24 07:51:07 pm

Aruki: 2016-01-24 07:46:07 pm

Aruki: 2015-06-05 02:14:28 am

Aruki: 2015-06-05 02:13:15 am

Aruki: 2015-06-05 01:23:36 am

Aruki: 2015-06-05 01:23:15 am

Aruki: 2015-05-30 11:46:29 am

Aruki: 2015-03-02 02:40:04 pm

Aruki: 2015-02-19 11:41:19 pm

Aruki: 2015-01-28 12:36:56 pm

Aruki: 2015-01-16 03:09:55 am

Aruki: 2014-10-05 10:33:17 pm

Aruki: 2014-10-05 10:33:12 pm

Aruki: 2014-10-05 01:30:01 pm

Aruki: 2014-10-05 01:28:19 pm

Aruki: 2014-09-09 06:05:15 pm

Aruki: 2014-09-09 07:41:57 am

Aruki: 2014-09-09 04:02:54 am

Aruki: 2014-08-26 11:24:37 pm

Aruki: 2014-08-21 12:12:20 am

Aruki: 2014-07-25 03:45:06 pm

Aruki: 2014-07-25 03:43:39 pm

Aruki: 2014-05-14 02:57:20 am

2014-05-14 02:40:08 am

Current tools:

Prime World Editor (Aruki): Level editor that supports the Metroid Prime trilogy and Donkey Kong Country Returns. Check the thread for the most up-to-date info.
URDE/PathShagged/HECL (Antidote/jackoalan): Collection of tools by Antidote and jackoalan that export a lot of the game's data to more common formats. Supports extraction of levels and rigged/animated character models, among other things. Its end goal is to reimplement the original game's engine as closely as possible. This is the best tool to use for extracting models/animations/etc from the game so far.
PakTool (Aruki): Can extract files from paks, as well as rebuild paks that can be used ingame. Extracted files are automatically decompressed and repacked files can be recompressed. Is capable of decompressing/recompressing MREA files as well as Tropical Freeze's formats with compressed data. Can also dump out a list of files contained in the pak. Fully supports the Prime trilogy and Donkey Kong Country Returns, and can extract and decompress files from Tropical Freeze.
Audacity GameCube ADPCM (jackoalan): Audacity fork that supports importing and exporting almost every standalone audio format from every Retro game. Doesn't support AGSC, ADP, or RSF yet.
STRG Editor (Aruki): Reads/writes STRG files (string tables) from both Prime 1 and Echoes, in all seven languages supported by the games. Does not support Corruption or DKCR.

Outdated tools - these are here in case you need them, but for the most part there are better ones available now:

CMDL Maxscript (Aruki): Imports CMDLs from Metroid Prime 1/2 into 3DS Max. Also handles CMPR TXTR conversion. Other texture formats are unsupported. You should probably use HECL instead.
MREA Maxscript (Aruki): Imports MREAs from Metroid Prime 1 into 3DS Max. Comes with the same TXTR functionality as the CMDL script. You should probably use HECL instead.
Prime Audio Decoder (Aruki): Extracts and decodes music and sound effects, and converts them to wav. Compatible with AGSC, CSMP, and DSP (Prime 1 only). You should probably use jackoalan's Audacity branch instead unless you're interested in ripping AGSC files.

Original post:

This board is much more active than the hacks and fangames board, so.

I've been doing some research on various file formats in Prime recently and written a couple tools to parse some of them. I would really love to be able to pull everything together well enough to be able to write a comprehensive viewer/editor, but I'm new to programming so I'm not totally sure how far I'll be able to get with this.

So what I've managed so far is:

- I've made my own tool to dump, decompress, and repack .pak files.
- I wrote some Maxscripts (3DS Max scripts) that can import geometry from CMDL and MREA files. There's still a couple important pieces of the format I'm missing so it's not super reliable and most non-trivial models crash it, but I've made good progress on it.
- Those scripts can also convert TXTR files to DDS.

Here's Furnace in 3DS Max, one of the only decently large rooms I've managed to load 100% correctly so far:

Miles has figured out a lot of stuff related to the object layout data in MREA files, as well as MLVL (contains metadata that defines a bunch of stuff related to how areas load - like which files to load for each room or which model to load as the skybox). He did this:

So yep, fun stuff. Getting somewhere with all this. The main goal at the moment is to write a viewer, then an editor, for MREA files and their associated object layout data.

Thread title:

[b]Current tools:[/b]
[list][li][url=https://m2k2.taigaforum.com/post/prime_world_editor_v1.0.1.html][b]Prime World Editor[/b] (Aruki)[/url]: Level editor that supports the Metroid Prime trilogy and Donkey Kong Country Returns. Check the thread for the most up-to-date info.[/li]
[li][url=https://m2k2.taigaforum.com/post/urde.html][b]URDE/PathShagged/HECL[/b] (Antidote/jackoalan)[/url]: Collection of tools by Antidote and jackoalan that export a lot of the game's data to more common formats. Supports extraction of levels and rigged/animated character models, among other things. Its end goal is to reimplement the original game's engine as closely as possible. This is the best tool to use for extracting models/animations/etc from the game so far.[/li]
[li][url=https://www.dropbox.com/s/bba2n9jzj719by4/PakTool.rar?dl=0][b]PakTool[/b] (Aruki)[/url]: Can extract files from paks, as well as rebuild paks that can be used ingame. Extracted files are automatically decompressed and repacked files can be recompressed. Is capable of decompressing/recompressing MREA files as well as Tropical Freeze's formats with compressed data. Can also dump out a list of files contained in the pak. Fully supports the Prime trilogy and Donkey Kong Country Returns, and can extract and decompress files from Tropical Freeze.[/li]
[li][url=https://github.com/jackoalan/audacity/releases][b]Audacity GameCube ADPCM[/b] (jackoalan)[/url]: Audacity fork that supports importing and exporting almost every standalone audio format from every Retro game. Doesn't support AGSC, ADP, or RSF yet.[/li]
[li][url=https://drive.google.com/file/d/0B9MLV21H7SDvSjBIY3BKRkQyZlk/edit?usp=sharing][b]STRG Editor[/b] (Aruki)[/url]: Reads/writes STRG files (string tables) from both Prime 1 and Echoes, in all seven languages supported by the games. Does not support Corruption or DKCR.[/li][/list]
[b]Outdated tools - these are here in case you need them, but for the most part there are better ones available now:[/b]
[list][li][url=https://drive.google.com/file/d/0B9MLV21H7SDvYXRtTWlFRGQ3ZjA/edit?usp=sharing][b]CMDL Maxscript[/b] (Aruki)[/url]: Imports CMDLs from Metroid Prime 1/2 into 3DS Max. Also handles CMPR TXTR conversion. Other texture formats are unsupported. You should probably use HECL instead.[/li]
[li][url=https://drive.google.com/file/d/0B9MLV21H7SDvMk9PN1lrM1g3ZVE/view?usp=sharing][b]MREA Maxscript[/b] (Aruki)[/url]: Imports MREAs from Metroid Prime 1 into 3DS Max. Comes with the same TXTR functionality as the CMDL script. You should probably use HECL instead.[/li]
[li][url=https://drive.google.com/file/d/0B9MLV21H7SDvemgwN1daYnliSjA/view?usp=sharing][b]Prime Audio Decoder[/b] (Aruki)[/url]: Extracts and decodes music and sound effects, and converts them to wav. Compatible with AGSC, CSMP, and DSP (Prime 1 only). You should probably use jackoalan's Audacity branch instead unless you're interested in ripping AGSC files.[/li][/list]

Original post:

[spoiler]This board is much more active than the hacks and fangames board, so.

I've been doing some research on various file formats in Prime recently and written a couple tools to parse some of them. I would really love to be able to pull everything together well enough to be able to write a comprehensive viewer/editor, but I'm new to programming so I'm not totally sure how far I'll be able to get with this.

So what I've managed so far is:

- I've made my own tool to dump, decompress, and repack .pak files.
- I wrote some Maxscripts (3DS Max scripts) that can import geometry from CMDL and MREA files. There's still a couple important pieces of the format I'm missing so it's not super reliable and most non-trivial models crash it, but I've made good progress on it.
- Those scripts can also convert TXTR files to DDS.

Here's Furnace in 3DS Max, one of the only decently large rooms I've managed to load 100% correctly so far:

[img]https://pbs.twimg.com/media/Bnd3N-UCYAAywqj.jpg:large[/img]

Miles has figured out a lot of stuff related to the object layout data in MREA files, as well as MLVL (contains metadata that defines a bunch of stuff related to how areas load - like which files to load for each room or which model to load as the skybox). He did this:

[img]https://pbs.twimg.com/media/BnkEAqfIMAIEFhN.png:large[/img]

So yep, fun stuff. Getting somewhere with all this. The main goal at the moment is to write a viewer, then an editor, for MREA files and their associated object layout data.[/spoiler]

Hazel

Edit history:

Hazel: 2014-05-14 08:06:05 am

2014-05-14 08:05:04 am

rocks, locks, and invisible blocks

Metroid Prime 2 / 3's maps imported and scaled to work in Metroid Prime when?
no but this is actually really interesting stuff you guys have going

MilesSMB

2014-05-14 09:12:42 am

When you figure out how to decompress Prime 2 and 3's room files.

kirbymastah

2014-05-14 11:16:24 am

~~and hunters?~~

Aruki

Edit history:

Aruki: 2014-05-14 02:17:28 pm

2014-05-14 02:07:01 pm

I am totally going to import Grand Abyss into MP1.

MrSpEeDrUn

Edit history:

MrSpEeDrUn: 2014-05-14 06:28:53 pm

2014-05-14 06:28:43 pm

MP2 also a bit MP1 speedrunning

Sound's like fun. I always wanted to make my own mp1 romhack.

ArXyn

2014-05-15 06:44:51 am

This is exciting.

Quietus

2014-05-15 08:50:01 am

Super Secret Area - Dead Ahead!

I don't know whether there's anything here you can make use of. Think

MilesSMB

2014-05-15 09:04:05 am

Parax is the last person to have posted there (a few months ago). <_<

Ntsc

Edit history:

Ntsc: 2014-05-17 02:54:13 pm

Ntsc: 2014-05-17 02:51:34 pm

2014-05-17 02:49:08 pm

You should see if MetroidMenace is still around. He was actually the reason I even joined MP2k. I don't know if he ever released source.

On a related note if you want help you really should setup a github repo for your work.
I've seen literally hundreds of interesting ideas die simply because the original programmer got bored and never released the source.

Here is one of the few images MetroidMenace made that still exists. He also showed the Landing Site rendering pretty well ~~but I can't find the image anymore.~~

Edit: Oh yeah I forgot about this post

https://m2k2.taigaforum.com/post/ped_suit_progression.html

Aruki

Edit history:

Aruki: 2014-05-18 03:38:46 am

Aruki: 2014-05-17 07:54:33 pm

2014-05-17 06:39:13 pm

Neat.

I'm not really looking for help at the moment and I'd be embarrassed to release my code as-is (at least until I've learned a bit more... the stuff I know right now is horribly basic), so I think I'll hold onto it for now. I don't anticipate going anywhere anytime soon though so if anyone else wants to try to do something similar in the future and wanted any help I'd likely be around.

That looks impressive though, going to be a challenge to make something up to that quality level. Actually looks fairly similar to the sort of comprehensive viewer/editor I had in mind.

I'm learning some basic OpenGL stuff at the moment, managed to get the geometry for Furnace rendered correctly but haven't implemented proper texture support yet.

edit: success, woo

Ntsc

Edit history:

Ntsc: 2014-05-20 05:40:08 pm

Ntsc: 2014-05-18 11:30:52 am

Ntsc: 2014-05-18 11:00:52 am

2014-05-18 10:44:56 am

Don't be embarrassed. The open source community is probably the nicest community on the internet. Your program looks great. It almost looks like you are just missing shader/lighting support.

I really do recommend putting all of your scripts and other source into git repo - even if it is just on your local machine. If you decide to open source it you'll be that much closer and even if you don't you'll get all the benefits of source control (branching to test new features, history to see why you wrote something that way, and most importantly the ability to never lose anything by accidentally deleting a file).

I open source nearly everything because honestly I can't finish anything.

Aruki

Edit history:

Aruki: 2014-05-18 10:02:23 pm

Aruki: 2014-05-18 09:37:23 pm

Aruki: 2014-05-18 03:50:02 pm

Aruki: 2014-05-18 03:47:27 pm

Aruki: 2014-05-18 03:46:54 pm

2014-05-18 03:46:33 pm

Nah there's a lot more missing than that. I wanna try to make some sort of GUI + material editor first because I really need to understand the material format better to be able to reliably load meshes (I'm pretty sure a mesh's vertex size is determined by what texture units are on its material). The reason I'm testing on Furnace is because it's one of the only rooms that I know actually works at the moment.

If I can get that and get it to where I can load any CMDL/MREA consistently though, there should be nothing stopping me from loading objects in.

Anyway, I guess I'll look into git later. I'm still not interested whatsoever in open sourcing anything right now, but I suppose the backup functionality sounds useful.

edit: on the subject of lighting - the game uses a lot of baked lighting in the form of lightmaps, so theoretically you could load them into an external program and it'd look pretty close to how it does ingame :P

edit 2: hah, this actually looks pretty cool. I disabled UV maps in the viewer and I ended up getting cel-shaded Metroid Prime:

Nah there's a lot more missing than that. I wanna try to make some sort of GUI + material editor first because I really need to understand the material format better to be able to reliably load meshes (I'm pretty sure a mesh's vertex size is determined by what texture units are on its material). The reason I'm testing on Furnace is because it's one of the only rooms that I know actually works at the moment.

If I can get that and get it to where I can load any CMDL/MREA consistently though, there should be nothing stopping me from loading objects in.

Anyway, I guess I'll look into git later. I'm still not interested whatsoever in open sourcing anything right now, but I suppose the backup functionality sounds useful.

edit: on the subject of lighting - the game uses a lot of baked lighting in the form of lightmaps, so theoretically you could load them into an external program and it'd look pretty close to how it does ingame :P

edit 2: hah, this actually looks pretty cool. I disabled UV maps in the viewer and I ended up getting cel-shaded Metroid Prime:

[img]https://pbs.twimg.com/media/Bn9dyUACUAE3fNU.png:large[/img]

ArXyn

Edit history:

ArXyn: 2014-05-19 02:14:05 pm

ArXyn: 2014-05-19 02:14:03 pm

2014-05-19 02:12:51 pm

Step 1) Release a full hack where the only difference is adding this art.
Step 2) Profit

kirbymastah

2014-05-19 03:23:00 pm

Quote from ArXyn:

Step 1) Get rid of chozo ghosts in this room
Step 2) Release a full hack where the only difference is adding this art.
Step 3) Profit

Fixed

DJGrenola

2014-05-19 03:45:14 pm

Quote from MilesSMB:

When you figure out how to decompress Prime 2 and 3's room files.

some years ago now SkippyJr disassembled part of the prime 2 loader code, and he told me it used segmented LZO1X.

Aruki

2014-05-19 07:10:01 pm

Yeah, MP2 uses LZO. The MREAs are weird though, because they aren't marked as compressed, and the file itself isn't, but a lot of separate chunks of the file actually are. Miles did figure it out a little while ago though... apparently there's a table right after the header that says what parts of the file are compressed and each of those sections needs to be decompressed separately. Haven't looked at it myself but that's what he's told me.

Ntsc

2014-05-20 05:44:20 pm

I'm home sick today - found this which may be helpful https://code.google.com/p/mpxviewer. I didn't see any links to it on metroid construction.

Aruki

2014-05-20 09:24:11 pm

Yeah, intrdpth actually has a thread on here about MpxViewer somewhere. I've looked through the code for it already, used parts of it as a reference to get started. He doesn't read the vertex sizes properly either... he does some hacks while reading the primitive data to find out the size of each one. IIRC a few rooms crash it too so it's certainly not an optimal solution.

MilesSMB

2014-05-21 10:37:03 am

So I streamed my first attempt at a Prime hack last night. It's a pickup randomizer, shuffling around the positions of most items in the game. You can check the pastebin I had during the stream for more info: http://pastebin.com/ZuGxP3C2

It wasn't possible to complete (nor was it intended to be at the moment), but it was still a ton of fun, and certainly had some awesome moments. :P

kirbymastah

2014-05-22 01:15:55 am

Quote from MilesSMB:

So I streamed my first attempt at a Prime hack last night. It's a pickup randomizer, shuffling around the positions of most items in the game. You can check the pastebin I had during the stream for more info: http://pastebin.com/ZuGxP3C2

It wasn't possible to complete (nor was it intended to be at the moment), but it was still a ton of fun, and certainly had some awesome moments. :P

I'm wondering why you never bothered with grappling through magmoor in your first trip

ArXyn

2014-05-22 05:29:31 am

I heard it isn't possible to use the grapple beam in heated rooms without varia.

kirbymastah

2014-05-22 09:43:46 am

Quote from ArXyn:

I heard it isn't possible to use the grapple beam in heated rooms without varia.

oh right, forgot about that

MrSpEeDrUn

2014-05-22 03:55:38 pm

MP2 also a bit MP1 speedrunning

awesome, watched the recording, though couldn't you IS the item behind that tree super missile thing in chozo ruins?
I think you'd have enough energy for that.

MilesSMB

Edit history:

MilesSMB: 2014-05-22 05:06:17 pm

2014-05-22 05:06:03 pm

I can't tell if Kirby is joking or not.

Quote from MrSpEeDrUn:

awesome, watched the recording, though couldn't you IS the item behind that tree super missile thing in chozo ruins?
I think you'd have enough energy for that.

I remember having the game crash when trying to IS in Main Plaza before, though that was a long time ago so who knows. Anyway there were several expansions I could've tried for, but every suit and Wave ended up being in the Mines so it was impossible to complete.