Metroid Prime hacking research

2014-12-08 10:37:33 am

Really? I guess you're right though, after comparing both rui_flaahgraL and rui_flaahgraR (assuming those are the audio files for it) the NTSC version is 663kb while the PAL version is 2.1mb.

And how would someone go along adding some of the cool things that that PAL version got such as the different main menu intro (this one's probably in the Video folder) and the grapple beam cutscene?

Aruki

2014-12-08 01:45:52 pm

The main menu animations are just video files, so you can probably just swap out with the PAL ones for that as well. Changing ingame cutscenes would require you to actually modify the MREA files in the paks though, as well as transfer any new files in the PAL version associated with the cutscene like animations/character sets/particles/sounds/etc. Would be a big headache.

Antidote

Edit history:

Antidote: 2014-12-08 04:44:57 pm

2014-12-08 04:41:31 pm

Yeah, all of MP's cutscenes are live. The only thing retro pre-rendered are the "attractions" and menu animations.
On top of that, retro uses MTHP, which have no audio, so keep that in mind when shuffling those around.

aminemo

Edit history:

aminemo: 2014-12-13 04:09:10 pm

aminemo: 2014-12-13 04:09:04 pm

aminemo: 2014-12-13 04:08:34 pm

aminemo: 2014-12-13 02:52:05 pm

2014-12-13 02:51:18 pm

Hello Guys
I am currently trying to help the dolphin Vr dev team by finding an AR code or another hack which can remove the culling from the game
because the geometry disappear when you turn your head outside of the main rendering viewport.

I got a metroid demo and find a 3 differents .map files containing some interesting functions.
I am trying to work with AR codes but which .map file refering to the .dol ?
There is three different versions ..I am confused.
Somebody can help?

Aruki

Edit history:

Aruki: 2014-12-13 03:37:13 pm

Aruki: 2014-12-13 03:31:37 pm

Aruki: 2014-12-13 03:31:19 pm

Aruki: 2014-12-13 03:30:34 pm

2014-12-13 03:20:24 pm

Already replied in your thread, but there really isn't a need to post this twice in different places.

In other news, thanks to Ninji/Treeki and the demo disassembly, material animations are partially cracked. The thing I thought might be "keyframes" are actually different animation types. The first u32 value is a number 0-7 that specifies the animation type, and following that are a couple float parameters. The number of parameters depends on the type. The known types are 2/3/4/5; I don't know what the others do, exactly. They're weird. Here's how they work:

s = seconds mod 900

Mode 2: UV offset
float offsetA, offsetB, scaleA, scaleB;
uOffset = (s * scaleA) + offsetA;
vOffset = (s * scaleB) + offsetB;

Mode 3: Rotation
float offset, scale;
angle = (s * scale) + offset;
Material is then rotated by [angle] degrees.

Mode 4/5: U offset and V offset, respectively. Four float parameters, but I don't know how it calculates this yet, exactly; it's rather confusing. That said, it actually uses the same code for both these modes, so it calculates the same offset either way; it just changes whether it's applied to U or V.

Like I've said, I don't know what the other four modes do, but I think they take no parameters. I'm also not sure exactly how different types are mixed and matched, since it's possible to specify more than one per material.

Already replied in [url=https://m2k2.taigaforum.com/post/metroid_prime_hacking_help.html]your thread[/url], but there really isn't a need to post this twice in different places.

In other news, thanks to Ninji/Treeki and the demo disassembly, material animations are partially cracked. The thing I thought might be "keyframes" are actually different animation types. The first u32 value is a number 0-7 that specifies the animation type, and following that are a couple float parameters. The number of parameters depends on the type. The known types are 2/3/4/5; I don't know what the others do, exactly. They're weird. Here's how they work:

s = seconds mod 900

Mode 2: UV offset
float offsetA, offsetB, scaleA, scaleB;
uOffset = (s * scaleA) + offsetA;
vOffset = (s * scaleB) + offsetB;

Mode 3: Rotation
float offset, scale;
angle = (s * scale) + offset;
Material is then rotated by [angle] degrees.

Mode 4/5: U offset and V offset, respectively. Four float parameters, but I don't know how it calculates this yet, exactly; it's rather confusing. That said, it actually uses the same code for both these modes, so it calculates the same offset either way; it just changes whether it's applied to U or V.

Like I've said, I don't know what the other four modes do, but I think they take no parameters. I'm also not sure exactly how different types are mixed and matched, since it's possible to specify more than one per material.

Sunbutt

2014-12-14 06:16:33 pm

Okayy so after so long of disappearing for the most part - I've been lurking and watching every now and then - I've been able to get the money to pay of 3DS Max 2015. Granted, unfortunately I found no option to buy '12.

I'm not too sure how far models and animations have come, and whether either or both can be exported to any other game; alot of this technical talk has been confusing. Based on what I'm seeing now, it looks like animations is in the process of being broken through.

Aruki

Edit history:

Aruki: 2014-12-14 07:17:21 pm

2014-12-14 06:33:05 pm

We're at the point where importing skinned models is feasible, but there's no tools to do it just yet. Not sure anyone's tried cracking animations yet. There are two relevant formats - ANIM and EVNT. I'm not sure on this, but I believe ANIM contains the regular bone transform data and EVNT contains data on particle/sound/etc placement/timing. I'm interested in figuring it out eventually but we need skinned models first; otherwise there isn't really much of a way to test that it's coming out correctly.

EDIT: oh, just to clarify, the material animations I was talking about in my post above are unrelated to model/bone animations; they're for materials, like scrolling or rotating textures.

Antidote

2014-12-16 02:58:15 pm

I've been looking at ANIM with not much luck

Bearborg

2014-12-16 04:00:27 pm

Is there any chance of PakTool supporting paks from the Prime 3 E3 Demo? I imagine there's some interesting stuff in those.

Aruki

2014-12-16 06:31:11 pm

No. But there is an alternative version of the program specifically for that pak format.

Bearborg

2014-12-16 08:39:26 pm

Awesome, thanks.

MrSinistar

2014-12-17 09:31:38 pm

Quick miscellaneous tidbit: I was messing around with the sound effects for Metroid Prime 2 and I found out that the series of beeps that plays after you finish scanning an object, isn't just one sound file, but rather 3 sound effects being played at the same time. LOL, why the hell didn't they mix down the beeping rather than having it sequenced in-game?

Anyways, again, excellent work guys! I can't wait to get my hands on the editor!

Aruki

Edit history:

Aruki: 2015-01-03 02:15:00 am

2015-01-03 02:14:47 am

I realized I never actually posted any STRG documentation here, so here you go

Code:

struct strg
{
  u32 magic; // 0x87654321
  u32 version; // Prime: 0, Echoes: 1, Corruption: 3
  u32 language_count;
  u32 string_count;

  struct language {
    char lang_fourCC[4];
    u32 lang_stringtable_offset; // Relative to the start of the string tables
    u32 lang_stringtable_size; // Echoes only
  };
  std::vector<language> language_table;

  // Name table is Echoes-only
  u32 name_count;
  u32 name_table_size;
  struct name {
    u32 name_offset; // Relative to after name_table_size
    u32 string_index;
  };
  std::vector<name> name_table;
  std::vector<std::string> names; // Names are case-sensitively sorted in alphabetical order

  struct string_table {
    u32 table_size; // Prime 1 only
    std::vector<u32> string_offsets; // One value per string, obviously
    std::vector<std::wstring> strings;
  };
  std::vector<string_table> string_tables; // One string table per language
};

It is a simple format and it's been 100% cracked for a while. Corruption's STRG format is different, haven't bothered looking at it. Sorry for the somewhat messy notes.

I realized I never actually posted any STRG documentation here, so here you go

[code]struct strg
{
  u32 magic; // 0x87654321
  u32 version; // Prime: 0, Echoes: 1, Corruption: 3
  u32 language_count;
  u32 string_count;

struct language {
    char lang_fourCC[4];
    u32 lang_stringtable_offset; // Relative to the start of the string tables
    u32 lang_stringtable_size; // Echoes only
  };
  std::vector<language> language_table;

// Name table is Echoes-only
  u32 name_count;
  u32 name_table_size;
  struct name {
    u32 name_offset; // Relative to after name_table_size
    u32 string_index;
  };
  std::vector<name> name_table;
  std::vector<std::string> names; // Names are case-sensitively sorted in alphabetical order

struct string_table {
    u32 table_size; // Prime 1 only
    std::vector<u32> string_offsets; // One value per string, obviously
    std::vector<std::wstring> strings;
  };
  std::vector<string_table> string_tables; // One string table per language
};[/code]

It is a simple format and it's been 100% cracked for a while. Corruption's STRG format is different, haven't bothered looking at it. Sorry for the somewhat messy notes.

eth0s

Edit history:

eth0s: 2015-01-04 12:16:29 am

2015-01-04 12:15:48 am

sudo rm -rf /

Hey guys, new m2k2 user here.

I stumbled upon a Twitch VOD of Miles' where he showed off the Metroid Prime Randomizer mod that I believe he created, and that completely lit off lightbulbs in my head. I love to program and just recently graduated with a CS degree, but I have nothing spectacular to show for it; basically, I've been wanting to work on a bunch of cool personal projects, but for months I've been having trouble coming up with good ideas that I could follow up and implement into something awesome, until now.

Obviously I have a lot of research and catching up to do, but I would love to contribute to this in any way that I can. I'll be doing my best to search and read through as much info on modding the game as I can, but if you guys have any suggestions, they would be greatly appreciated.

Really impressive stuff you guys have been putting together here. Keep up the good work!

Aruki

2015-01-04 12:32:32 am

Hey eth0s, welcome to the forum! Are you interested in trying to make a mod, or in reverse engineering, or both? If you're interested in the RE aspect, there's a few Prime 1 formats left that haven't been cracked just yet, if you feel like taking a look at any of those:

- AFSM - finite state machine, used for AI
- ANIM - animations
- EVNT - animation POI data; I believe this is timing for things like particles and sound effects
- PATH - should be spline path data
- DPSC - collision response data
- CSNG - midi data, not sure what it's used for
- all of the particle formats - PART, ELSC (electric), SWHC (swoosh)

There's also half of AGSC (audio groups) - we can extract sounds from them, but we have no idea what the first half of the file is for.

I also don't think Echoes and Corruption have been researched as extensively as Prime 1; Echoes shares a lot of similarities with Prime 1, but Corruption has quite a few differences; lots of new formats and most of the old ones have been changed to some degree.

Aside from that, the information we have is kinda scattered around the thread right now, so if you want to know anything in particular feel free to ask in the thread or send one of us a PM.

Hey eth0s, welcome to the forum! Are you interested in trying to make a mod, or in reverse engineering, or both? If you're interested in the RE aspect, there's a few Prime 1 formats left that haven't been cracked just yet, if you feel like taking a look at any of those:

- AFSM - finite state machine, used for AI
- ANIM - animations
- EVNT - animation POI data; I believe this is timing for things like particles and sound effects
- PATH - should be spline path data
- DPSC - collision response data
- CSNG - midi data, not sure what it's used for
- all of the particle formats - PART, ELSC (electric), SWHC (swoosh)

There's also half of AGSC (audio groups) - we can extract sounds from them, but we have no idea what the first half of the file is for.

I also don't think Echoes and Corruption have been researched as extensively as Prime 1; Echoes shares a lot of similarities with Prime 1, but Corruption has quite a few differences; lots of new formats and most of the old ones have been changed to some degree.

Aside from that, the information we have is kinda scattered around the thread right now, so if you want to know anything in particular feel free to ask in the thread or send one of us a PM.

eth0s

2015-01-04 12:46:35 am

sudo rm -rf /

Hey Paraxade, thanks for replying!

I'm definitely interested in both, though the reverse engineering is definitely going to be significantly harder from the sounds of it, but hey, who doesn't like a challenge? It's awesome that you guys have been able to decrypt so much of the game as it is.

Is there a M2k2 IRC channel by any chance?

Aruki

2015-01-04 12:49:35 am

Uhh, there's a couple channels on irc.speeddemosarchive.com; the main channel is #metroid, which has been dead for years, and then there's #MetroidPrime, which Miles started more recently and might be more active. I haven't been in either in ages so I don't really know what the current state of them is. Could hop on right now if you want to talk about anything.

coppro

2015-01-05 02:53:33 am

There's also a general metroid-related channel #metroid on irc.lizardirc.net. There is some talk of hacking in there on occasion, and I also mention it to maybe get some more lurkers from this community in there.

(I'm still bugging my friend on writing out the DSP files, btw)

Lord Fredrik

2015-01-06 08:51:32 pm

Hi! I was doing some research about ripping DKCR models for a project of mine, and I stumbled across this thread just like that when extracting CMDL files from DKCR's PAK files.

I just want to know is it possible to view / import them in 3DS max yet?

Aruki

2015-01-07 12:42:03 am

I finally nailed down some really accurate transparency in PWE and I'm extremely happy about it! Have an awesome screenshot.

As far as I can tell, the depth sorting is extremely accurate. In 3D applications, transparent geometry has to be rendered in order from furthest to the camera to closest in order to look correct, which can be tough to pull off well, but I believe I've replicated the game's method for doing this. The process for that is:

- split meshes up by submesh
- take the bounding box of each submesh and take the coordinate of the vertex closest to the camera's direction vector
- calculate the distance of that vertex from the camera
- sort using the dot product of the distance vector and the camera's direction

This method even replicates a sorting bug. In the top room with the first Metroid, there's an orange monitor hanging over a glass window. If you look at it from dead on, it looks fine. But if you tilt the camera to the left or right, it gets layered behind the glass instead of over it. This bug happens both in the original game and in PWE. Yay accuracy!

I've been trying for a very long time to figure this out so I'm very happy to have it now. :P There's no objects in this screenshot because I still need to reimplement SCLY loading with my new new backend. There's tons more transparent stuff with the objects loaded (Metroids, glass tubes, Space Pirate scythes) so I can take another picture when that gets redone if you guys wanna see.

It's been going slow because I've been doing another large rewrite, but now most of it's finished and for once I finally feel like I have a good base to work from, so hopefully progress will start to come a bt quicker now.

Quote from Lord Fredrik:

Hi! I was doing some research about ripping DKCR models for a project of mine, and I stumbled across this thread just like that when extracting CMDL files from DKCR's PAK files.

I just want to know is it possible to view / import them in 3DS max yet?

Yeah, I wrote a modified version of my MP1/2/3 CMDL script a while ago to import DKCR models. I'll attach it to this post so you can use it. Bear in mind a couple things - first, a lot of materials import incorrectly due to my lack of understanding the MP3/DKCR material format; that said, the script should convert most of the textures the model uses, so you can fix it yourself by changing what textures are assigned to each material. The other thing to bear in mind is that this is a kinda messy modification of a script that was already kinda messy to begin with, and I'm not gonna bother going over it to clean it up right now, so the source is kinda screwy. Also, you can flip off the "weldverts" variable at the top of the script if you want faster imports at the cost of worse geometry. Here you go, though.

I finally nailed down some really accurate transparency in PWE and I'm extremely happy about it! Have an awesome screenshot.

[img]http://i.imgur.com/Wga8gxj.jpg[/img]

As far as I can tell, the depth sorting is extremely accurate. In 3D applications, transparent geometry has to be rendered in order from furthest to the camera to closest in order to look correct, which can be tough to pull off well, but I believe I've replicated the game's method for doing this. The process for that is:

- split meshes up by submesh
- take the bounding box of each submesh and take the coordinate of the vertex closest to the camera's direction vector
- calculate the distance of that vertex from the camera
- sort using the dot product of the distance vector and the camera's direction

This method even replicates a sorting bug. In the top room with the first Metroid, there's an orange monitor hanging over a glass window. If you look at it from dead on, [url=http://i.imgur.com/pCr2avR.jpg]it looks fine[/url]. But if you tilt the camera to the left or right, [url=http://i.imgur.com/CmpqkG6.jpg]it gets layered behind the glass instead of over it[/url]. This bug happens both in the original game and in PWE. Yay accuracy!

I've been trying for a very long time to figure this out so I'm very happy to have it now. :P There's no objects in this screenshot because I still need to reimplement SCLY loading with my new new backend. There's tons more transparent stuff with the objects loaded (Metroids, glass tubes, Space Pirate scythes) so I can take another picture when that gets redone if you guys wanna see.

It's been going slow because I've been doing another large rewrite, but now most of it's finished and for once I finally feel like I have a good base to work from, so hopefully progress will start to come a bt quicker now.

[quote="Lord Fredrik"]Hi! I was doing some research about ripping DKCR models for a project of mine, and I stumbled across this thread just like that when extracting CMDL files from DKCR's PAK files.

I just want to know is it possible to view / import them in 3DS max yet?[/quote]

Yeah, I wrote a modified version of my MP1/2/3 CMDL script a while ago to import DKCR models. I'll attach it to this post so you can use it. Bear in mind a couple things - first, a lot of materials import incorrectly due to my lack of understanding the MP3/DKCR material format; that said, the script should convert most of the textures the model uses, so you can fix it yourself by changing what textures are assigned to each material. The other thing to bear in mind is that this is a kinda messy modification of a script that was already kinda messy to begin with, and I'm not gonna bother going over it to clean it up right now, so the source is kinda screwy. Also, you can flip off the "weldverts" variable at the top of the script if you want faster imports at the cost of worse geometry. Here you go, though.

MrSpEeDrUn

2015-01-07 03:57:17 am

MP2 also a bit MP1 speedrunning

maybe you can figure out why the glass doesnt break sometimes that way.

Lord Fredrik

2015-01-07 12:13:24 pm

Quote from Paraxade:

I finally nailed down some really accurate transparency in PWE and I'm extremely happy about it! Have an awesome screenshot.
Yeah, I wrote a modified version of my MP1/2/3 CMDL script a while ago to import DKCR models. I'll attach it to this post so you can use it. Bear in mind a couple things - first, a lot of materials import incorrectly due to my lack of understanding the MP3/DKCR material format; that said, the script should convert most of the textures the model uses, so you can fix it yourself by changing what textures are assigned to each material. The other thing to bear in mind is that this is a kinda messy modification of a script that was already kinda messy to begin with, and I'm not gonna bother going over it to clean it up right now, so the source is kinda screwy. Also, you can flip off the "weldverts" variable at the top of the script if you want faster imports at the cost of worse geometry. Here you go, though.

Hey, thanks for the help, mate. I'll try it out! ^_^

[quote="Paraxade"]I finally nailed down some really accurate transparency in PWE and I'm extremely happy about it! Have an awesome screenshot.
Yeah, I wrote a modified version of my MP1/2/3 CMDL script a while ago to import DKCR models. I'll attach it to this post so you can use it. Bear in mind a couple things - first, a lot of materials import incorrectly due to my lack of understanding the MP3/DKCR material format; that said, the script should convert most of the textures the model uses, so you can fix it yourself by changing what textures are assigned to each material. The other thing to bear in mind is that this is a kinda messy modification of a script that was already kinda messy to begin with, and I'm not gonna bother going over it to clean it up right now, so the source is kinda screwy. Also, you can flip off the "weldverts" variable at the top of the script if you want faster imports at the cost of worse geometry. Here you go, though.[/quote]

Hey, thanks for the help, mate. I'll try it out! ^_^

Aruki

2015-01-08 03:37:48 am

Quote from Lord Fredrik:

Quote from Paraxade:

I finally nailed down some really accurate transparency in PWE and I'm extremely happy about it! Have an awesome screenshot.
Yeah, I wrote a modified version of my MP1/2/3 CMDL script a while ago to import DKCR models. I'll attach it to this post so you can use it. Bear in mind a couple things - first, a lot of materials import incorrectly due to my lack of understanding the MP3/DKCR material format; that said, the script should convert most of the textures the model uses, so you can fix it yourself by changing what textures are assigned to each material. The other thing to bear in mind is that this is a kinda messy modification of a script that was already kinda messy to begin with, and I'm not gonna bother going over it to clean it up right now, so the source is kinda screwy. Also, you can flip off the "weldverts" variable at the top of the script if you want faster imports at the cost of worse geometry. Here you go, though.

Hey, thanks for the help, mate. I'll try it out! ^_^

sorry, probably should have at least checked to make sure it actually worked first. :P Use this one instead.

The script definitely has a lot of flaws, so I'll probably fix it up to work better once I start working more seriously with DKCR, though that may be a while. Honestly all of my maxscripts could do with a complete rewrite. Lots of really old code for things that I have a much better understanding of now. The Prime 1 MREA import script is a giant mess, that's the main reason I haven't released it yet.

For now it's all I got so it'll have to suffice until I write a better one (or someone else does).

[quote="Lord Fredrik"][quote="Paraxade"]I finally nailed down some really accurate transparency in PWE and I'm extremely happy about it! Have an awesome screenshot.
Yeah, I wrote a modified version of my MP1/2/3 CMDL script a while ago to import DKCR models. I'll attach it to this post so you can use it. Bear in mind a couple things - first, a lot of materials import incorrectly due to my lack of understanding the MP3/DKCR material format; that said, the script should convert most of the textures the model uses, so you can fix it yourself by changing what textures are assigned to each material. The other thing to bear in mind is that this is a kinda messy modification of a script that was already kinda messy to begin with, and I'm not gonna bother going over it to clean it up right now, so the source is kinda screwy. Also, you can flip off the "weldverts" variable at the top of the script if you want faster imports at the cost of worse geometry. Here you go, though.[/quote]

Hey, thanks for the help, mate. I'll try it out! ^_^[/quote]

sorry, probably should have at least checked to make sure it actually worked first. :P Use this one instead.

The script definitely has a lot of flaws, so I'll probably fix it up to work better once I start working more seriously with DKCR, though that may be a while. Honestly all of my maxscripts could do with a complete rewrite. Lots of really old code for things that I have a much better understanding of now. The Prime 1 MREA import script is a giant mess, that's the main reason I haven't released it yet.

For now it's all I got so it'll have to suffice until I write a better one (or someone else does).

attachment:

dkcr_cmdl.rar

Lord Fredrik

2015-01-09 06:48:22 pm

Yes, thank you. I found Tiki Tong and DK and they look better now than with the old script.

MrSinistar

2015-01-10 06:22:56 am

Dude, that's awesome that you got the transparency and depth sorting all figured out. Major kudos!!