Metroid Prime hacking research

Edit history:

Aruki: 2015-09-09 11:14:15 pm

2015-09-09 08:11:09 pm

If anyone's interested in DKCR 3D, I cracked the RST0 archive format + the compression it uses and wrote a tool to extract the files out. You can download it here: https://www.dropbox.com/s/fvxzkflshjn1rd5/RSTExtract.rar?dl=0

There's a ton of format differences compared to the original DKCR (which is probably not the most surprising thing ever, considering it's on different hardware). There's actually some pretty interesting stuff in there, too - it seems like there's more metadata than the original game, so there's stuff like filenames for every file and even something that looks like object parameter names. I'm not sure if the stuff I worked out should go on the wiki, since I don't think Retro did the port, but ehhh I don't see too many reasons why not?

edit: I went and wrote up what I figured out on the Wiki, so here it is

Lord Fredrik

2015-09-09 11:55:02 pm

I love when you're the one who bumps this thread lol. Always cool stuff.

wowsers

2015-09-10 02:59:17 pm

All I can do in programming is write shaders basically :(

Odylg

2015-09-10 03:41:57 pm

I run this here hotel of an evening

Quote from wowsers:

All I can do in programming is write shaders basically :(

Still better than I can do

Aruki

2015-09-10 10:21:34 pm

Something I found kinda amusing: The fact that DKCR 3D has filenames makes it more obvious when assets are reused. There are a bunch of sound effects that were obviously taken from Metroid Prime (mostly from 3, a couple from 1/2):

Quote:

05_bryyo_missilerack_retrac.sfx
aurora_ds_summon_lp.sfx
aurora_stunned_head_lp.sfx
bryyo_stone_lp.sfx
bryyo_stone_lpd1.sfx
bryyo_stone_stop.sfx
deathball_appear_1.sfx
deathball_appear_2.sfx
deathball_appear_3.sfx
ds_missile_fire_1.sfx
ds_missile_fire_2.sfx
e_garbeetle_attack7.sfx
e_garbeetle_attack8.sfx
e_garbeetle_hit_light3.sfx
e_garbeetle_hit_light4.sfx
e_pudtoad_dive1_edit.sfx
ghor_damage_1.sfx
ghor_damage_2.sfx
ghor_damage_3.sfx
ghor_jump_descend.sfx
korakk_foot_back_1.sfx
korakk_foot_back_2.sfx
korakk_foot_back_3.sfx
korakk_footrear_1.sfx
korakk_footrear_1a.sfx
mogenar_boot_destroyed_1.sfx
mogenar_boot_destroyed_1_v2.sfx
mogenar_boot_destroyed_2.sfx
mogenar_boot_destroyed_2_v2.sfx
reptilicus_walk_1_rev.sfx
reptilicus_walk_2_rev.sfx
reptilicus_walk_3_rev.sfx
reptilicus_walk_4_rev.sfx
rid_b_voxidle_00.sfx
rid_b_voxidle_01.sfx
rid_b_walk_00.sfx
ridley_intro_walk_1.sfx
ridley_intro_walk_2.sfx
ridley2_bomb_fall_1.sfx
ridley2_bomb_fall_2.sfx
ridley2_bomb_fall_3.sfx
ridley2_bomb_impact_1.sfx
ridley2_bomb_impact_2.sfx
ridley2_bomb_impact_3.sfx
ridley2_fireball_impact.sfx
rundas_jump_up.sfx
rundas_pillar_explode_2.sfx
trooper_hypermode_lp.sfx

Something I found kinda amusing: The fact that DKCR 3D has filenames makes it more obvious when assets are reused. There are a bunch of sound effects that were obviously taken from Metroid Prime (mostly from 3, a couple from 1/2):

[quote]05_bryyo_missilerack_retrac.sfx
aurora_ds_summon_lp.sfx
aurora_stunned_head_lp.sfx
bryyo_stone_lp.sfx
bryyo_stone_lpd1.sfx
bryyo_stone_stop.sfx
deathball_appear_1.sfx
deathball_appear_2.sfx
deathball_appear_3.sfx
ds_missile_fire_1.sfx
ds_missile_fire_2.sfx
e_garbeetle_attack7.sfx
e_garbeetle_attack8.sfx
e_garbeetle_hit_light3.sfx
e_garbeetle_hit_light4.sfx
e_pudtoad_dive1_edit.sfx
ghor_damage_1.sfx
ghor_damage_2.sfx
ghor_damage_3.sfx
ghor_jump_descend.sfx
korakk_foot_back_1.sfx
korakk_foot_back_2.sfx
korakk_foot_back_3.sfx
korakk_footrear_1.sfx
korakk_footrear_1a.sfx
mogenar_boot_destroyed_1.sfx
mogenar_boot_destroyed_1_v2.sfx
mogenar_boot_destroyed_2.sfx
mogenar_boot_destroyed_2_v2.sfx
reptilicus_walk_1_rev.sfx
reptilicus_walk_2_rev.sfx
reptilicus_walk_3_rev.sfx
reptilicus_walk_4_rev.sfx
rid_b_voxidle_00.sfx
rid_b_voxidle_01.sfx
rid_b_walk_00.sfx
ridley_intro_walk_1.sfx
ridley_intro_walk_2.sfx
ridley2_bomb_fall_1.sfx
ridley2_bomb_fall_2.sfx
ridley2_bomb_fall_3.sfx
ridley2_bomb_impact_1.sfx
ridley2_bomb_impact_2.sfx
ridley2_bomb_impact_3.sfx
ridley2_fireball_impact.sfx
rundas_jump_up.sfx
rundas_pillar_explode_2.sfx
trooper_hypermode_lp.sfx[/quote]

Antidote

2015-09-11 04:45:21 pm

You guys remember that video with MP2's Zero Suit Samus that Parax posted a while back? And how her hand hilariously hovered when she entered her ship?
Mystery solved!

wowsers

2015-09-11 05:14:28 pm

Was that arm placement in any way related to rigging because working with it there might have been easier?

Antidote

2015-09-11 05:22:23 pm

No idea honestly :/

Odylg

2015-09-11 08:10:32 pm

I run this here hotel of an evening

Is there any way to set the shading to "material" as default instead of "solid" being default?

Antidote

Edit history:

Antidote: 2015-09-12 12:26:50 am

Antidote: 2015-09-12 12:16:09 am

2015-09-12 12:15:31 am

No need to worry about that, p4 now does that automatically.
You'll need to re-extract with -f though, or simply delete and start a new extract.

EDIT:
Jack, if you're reading this, I'm currently on FreeBSD and it doesn't have a Skype client >.>

jackoalan

Edit history:

jackoalan: 2015-09-15 08:41:12 pm

jackoalan: 2015-09-14 06:46:57 pm

2015-09-14 06:37:36 pm

I've been experimenting with the lightmap UV layers in the MREA data. Interested to see if the 0xBABEDEAD lights can be leveraged to re-render MP's lighting at higher resolutions.

First, a BABEDEAD -> Blender Cycles lighting test:

Now, the ultimate test: Cycles Texture Baking (this is an OpenGL screenshot, fully interactive):

Blender-powered lightmap rendering will be part of HECL/PathShagged in the next prototype release!

Aruki

2015-09-14 06:47:29 pm

Ooh, nice. I know that the BABEDEAD lights don't include -all- the lights that were used to render the lightmaps but they definitely have most of them in there.

jackoalan

2015-09-14 06:49:10 pm

Yea, areas like the elevator rooms are only really lit at player-level; even though the ceiling seems to have some pre-rendered lights.

Aruki

Edit history:

Aruki: 2015-09-21 11:16:43 pm

Aruki: 2015-09-21 11:11:54 pm

Aruki: 2015-09-21 11:11:40 pm

2015-09-21 04:39:06 pm

DKCR is kinda stupid it turns out. Prime 2 and 3 (which use the same SCLY format) always write every object property to the file, but DKCR commonly just completely omits properties. I'm not completely sure why this is, but my guess is that every parameter has a value that it's initialized to by default, and DKCR just doesn't bother writing the property if its value is equal to the default setting. The ideal way to handle that would be if we had a list of every default parameter value in the game; then the editor could initialize properties to that value and display them on the UI like normal, and omit them from the file the same way DKCR does, and the user wouldn't need to be aware that this is even happening. Unfortunately since I don't really see where we could get a list from other than by doing a ton of digging through the dol, the other solution I can think of is to have a way to mark a property as enabled or disabled on the UI, and then use that to determine whether to write values back to the file. (Alternately, maybe the DKCR3D .val files have something relevant? I feel like I should crack that format at some point so I can take a closer look at it.)

As an aside, I've generated object templates for DKCR, so now DKCR script objects are displaying much nicer in the editor.

edit: found where the default parameters are set... it's the constructor of the struct that they read the properties into. The structs are created at the beginning of the load function. I pulled the property list and default values of everything in LightParameters and from what I've tested so far my guess seems to be holding up - none of the properties ever show up as the value they're initialized to in the struct constructor. If I verify that I can do the same thing for the non-struct properties then that means pulling the default values of everything should be doable... it would take a ton of work but it would be doable and I'll probably grind it out in that case.

edit 2: Yes, I can do the same thing for the non-struct properties. Cool. This'll be fun...

DKCR is kinda stupid it turns out. Prime 2 and 3 (which use the same SCLY format) always write every object property to the file, but DKCR commonly just completely omits properties. I'm not completely sure why this is, but my guess is that every parameter has a value that it's initialized to by default, and DKCR just doesn't bother writing the property if its value is equal to the default setting. The ideal way to handle that would be if we had a list of every default parameter value in the game; then the editor could initialize properties to that value and display them on the UI like normal, and omit them from the file the same way DKCR does, and the user wouldn't need to be aware that this is even happening. Unfortunately since I don't really see where we could get a list from other than by doing a ton of digging through the dol, the other solution I can think of is to have a way to mark a property as enabled or disabled on the UI, and then use that to determine whether to write values back to the file. (Alternately, maybe the DKCR3D .val files have something relevant? I feel like I should crack that format at some point so I can take a closer look at it.)

As an aside, I've generated object templates for DKCR, so now DKCR script objects are displaying much nicer in the editor.

[img]https://pbs.twimg.com/media/CPa79qUUkAAYVLs.png:large[/img]

edit: found where the default parameters are set... it's the constructor of the struct that they read the properties into. The structs are created at the beginning of the load function. I pulled the property list and default values of everything in LightParameters and from what I've tested so far my guess seems to be holding up - none of the properties ever show up as the value they're initialized to in the struct constructor. If I verify that I can do the same thing for the non-struct properties then that means pulling the default values of everything should be doable... it would take a ton of work but it would be doable and I'll probably grind it out in that case.

edit 2: Yes, I can do the same thing for the non-struct properties. Cool. This'll be fun...

Jesse

2015-09-24 01:03:21 pm

yay, giving up on hacking this game was the right choice after all. Btw I always see you working on this program but never see a release or smth like that. Why no release or source code so that other people can build it?

Aruki

Edit history:

Aruki: 2015-09-25 04:29:57 am

Aruki: 2015-09-24 05:04:28 pm

2015-09-24 04:59:52 pm

I prefer working on it myself for right now and it's still not in a release ready state yet. I'll hopefully get it to that point soon.

Also apparently certain properties are never excluded from the file even though they have valid defaults set (like the unknown long in EditorProperties). I have no idea why.

Also even structs that are common to multiple objects can have their properties' default values overridden on a case-by-case basis by the object. So yeah this is super fun so far. We've got 33/137 of the objects done so far.

wowsers

2015-09-27 04:43:34 pm

I can only imagine mixing together the flat shaded mode with AO and see that, I bet it would look amazing

Aruki

2015-09-28 02:56:52 am

I figured out why there's multiple light layers... they're all completely separate sets of lights and objects can pick and choose which one they're lit by. It's the last property in LightParameters. MP1 only has three rooms that have a second layer - Arboretum (unused), Sunchamber Access (used by the Reaper Vines), and Monitor Tunnel (used by the platforms). It seems like in the odd case where an object specifies an invalid/empty light layer index (like Lava Lake puffers), the game defaults it back to layer 0. Samus is lit by layer 0.

This applies to all the games. MP1/2 can have two light layers per area, MP3 can have four, DKCR can have eight.

Willphonse

2015-09-28 05:25:59 am

Hello?

Hey! Just wanted to drop by and ask a few things.
I've been following this thread for a while, and I've become real interested in this whole thing.

I'm mainly curious about the Prime World Editor at this point. Are we looking at a release this year? I'm super psyched for any kind of release whatsoever!
That said, is PWE similar in function to Unity?
And finally, Is there a list of things it'll be able to do? I'm interested in adding MP2 maps into MP1 and other misc stuff like that.
I'd do all this stuff without the editor, but I'm not only clueless on how I can add a map into MP1, I'm also too lazy to sort through every file on every game to find the areas I'm looking for.

Thanks for the tools you guys have made/shared! I can't wait to see how this all pans out!

Aruki

2015-09-28 06:07:58 am

I'm hoping for this year. The problem is that since I mainly post when I've made progress, the sheer amount of stuff there is left to do and the amount of issues in the current build of the editor aren't really visible. There are a lot of missing features and issues that need to be addressed. This is the gist of what's on my to-do list before the first release:

- A few property types need to be supported (enums, arrays, and bitflags).
- Editing properties needs to actually change the value of the property in the backend and reflect changes in the viewport (if applicable).
- I need to support rewriting MREA and MLVL files with changes, for all four supported games (MP1, MP2, MP3, DKCR).
- There needs to be a prefab system for creating new objects. This prefab system also needs to be able to recognize prefabs that are already in the game's cooked data. (This is very tricky, but I do have an idea of how to implement it.)
- Likewise, there also needs to be the ability to delete objects. This is more complicated than it sounds when you consider making it work with the undo system, needing to delete and then recreate links to other objects, etc.
- There needs to be a way to link objects together to send messages to each other, and break existing links.
- The "default value" thing I was talking about needs to be implemented for DKCR. I mentioned an easier way of doing this but since I know how to get the default values and I think that would be way better I'm going to try to do it that way.
- The UI for loading a world/selecting areas needs to be redone and added to the main editor window (it's a separate popup window at the moment and would more cleanly fit in on the main window UI somewhere).
- Improved collision rendering, because the way it's done right now (with the grey checkerboard) is incredibly ugly.
- Dynamic lights for DKCR maps. (Dynamic lighting in MP3 was just crossed off the to-do list today!)
- The dark style needs to be properly redone because although the current one looks pretty nice for the most part, it's something that I just threw together in five minutes by copying some code off Google. It's buggy and needs a proper reimplementation.
- Improved controls for showing/hiding objects.
- Rendering fixes for DKCR. Some materials show up as pitch black and I haven't looked into why yet. I'm also suspecting that there's data somewhere that controls render order and I'd like to look into that to see if I can improve depth sorting.
- There's various things on the UI that I'd like to improve the functionality of and make more intuitive to use.
- Various debugging, optimizations, and other minor feature additions

The goal of the first release is to make a strong viewer for the levels/models and to allow editing script objects in the main world editor. That means basically creating objects, modifying object properties, linking objects, creating and modifying prefabs, etc. and of course resaving them so you can actually run the changes in the game. It won't be able to create full levels from scratch - things like importing/exporting assets, moving full areas around, attaching areas to other areas, etc. isn't planned for the first release. There are a LOT of things that I would like it to be able to do in the future but I want to get it into a state where I feel you could actually do some work on it. What I'm thinking is ideally after that point I would be able to do more frequent smaller updates by focusing on one feature at a time. Getting it into a stable state and making it functional as an editor to begin with requires a ton of groundwork to be done as a base, but the good news is most of that groundwork is finished.

It sounds like a lot but I think getting it all done by the end of the year is feasible. Possibly early next year at the latest. Worst case if I end up not having the time to continue working on it, I'll release what I have and consider open sourcing... I have no intention of letting the project die or sitting on it for an extended period without working on it or releasing it.

I'm hoping for this year. The problem is that since I mainly post when I've made progress, the sheer amount of stuff there is left to do and the amount of issues in the current build of the editor aren't really visible. There are a lot of missing features and issues that need to be addressed. This is the gist of what's on my to-do list before the first release:

- A few property types need to be supported (enums, arrays, and bitflags).
- Editing properties needs to actually change the value of the property in the backend and reflect changes in the viewport (if applicable).
- I need to support rewriting MREA and MLVL files with changes, for all four supported games (MP1, MP2, MP3, DKCR).
- There needs to be a prefab system for creating new objects. This prefab system also needs to be able to recognize prefabs that are already in the game's cooked data. (This is very tricky, but I do have an idea of how to implement it.)
- Likewise, there also needs to be the ability to delete objects. This is more complicated than it sounds when you consider making it work with the undo system, needing to delete and then recreate links to other objects, etc.
- There needs to be a way to link objects together to send messages to each other, and break existing links.
- The "default value" thing I was talking about needs to be implemented for DKCR. I mentioned an easier way of doing this but since I know how to get the default values and I think that would be way better I'm going to try to do it that way.
- The UI for loading a world/selecting areas needs to be redone and added to the main editor window (it's a separate popup window at the moment and would more cleanly fit in on the main window UI somewhere).
- Improved collision rendering, because the way it's done right now (with the grey checkerboard) is incredibly ugly.
- Dynamic lights for DKCR maps. (Dynamic lighting in MP3 was [url=https://pbs.twimg.com/media/CP-VWObU8AAtSop.png:large]just crossed off the to-do list today![/url])
- The dark style needs to be properly redone because although the current one looks pretty nice for the most part, it's something that I just threw together in five minutes by copying some code off Google. It's buggy and needs a proper reimplementation.
- Improved controls for showing/hiding objects.
- Rendering fixes for DKCR. Some materials show up as pitch black and I haven't looked into why yet. I'm also suspecting that there's data somewhere that controls render order and I'd like to look into that to see if I can improve depth sorting.
- There's various things on the UI that I'd like to improve the functionality of and make more intuitive to use.
- Various debugging, optimizations, and other minor feature additions

The goal of the first release is to make a strong viewer for the levels/models and to allow editing script objects in the main world editor. That means basically creating objects, modifying object properties, linking objects, creating and modifying prefabs, etc. and of course resaving them so you can actually run the changes in the game. It won't be able to create full levels from scratch - things like importing/exporting assets, moving full areas around, attaching areas to other areas, etc. isn't planned for the first release. There are a LOT of things that I would like it to be able to do in the future but I want to get it into a state where I feel you could actually do some work on it. What I'm thinking is ideally after that point I would be able to do more frequent smaller updates by focusing on one feature at a time. Getting it into a stable state and making it functional as an editor to begin with requires a ton of groundwork to be done as a base, but the good news is most of that groundwork is finished.

It sounds like a lot but I think getting it all done by the end of the year is feasible. Possibly early next year at the latest. Worst case if I end up not having the time to continue working on it, I'll release what I have and consider open sourcing... I have no intention of letting the project die or sitting on it for an extended period without working on it or releasing it.

MrHostille

2015-10-01 07:34:55 am

Hello , I have problems with visual c ++ 2011 to compile the CMDL to obj.

Build started 01/10/2015 13:21:27.
1>Project "C:\Users\DJ Poizzon\Desktop\SD\Download\dkcr_cmdl_to_obj-master\dkcr_cmdl_to_obj-master\cmdl_parser\cmdl_parser.vcxproj" on node 2 (build target(s)).
Project file contains ToolsVersion="12.0". This toolset may be unknown or missing, in which case you may be able to resolve this by installing the appropriate version of MSBuild, or the build may have been forced to a particular ToolsVersion for policy reasons. Treating the project as if it had ToolsVersion="4.0". For more information, please see http://go.microsoft.com/fwlink/?LinkId=291333.
1>C:\Program Files\MSBuild\Microsoft.Cpp\v4.0\Platforms\Win32\Microsoft.Cpp.Win32.Targets(518,5): error MSB8008: Specified platform toolset (v120) is not installed or invalid. Please make sure that a supported PlatformToolset value is selected.
1>Done Building Project "C:\Users\DJ Poizzon\Desktop\SD\Download\dkcr_cmdl_to_obj-master\dkcr_cmdl_to_obj-master\cmdl_parser\cmdl_parser.vcxproj" (build target(s)) -- FAILED.

Build FAILED.

Time Elapsed 00:00:01.70.

I'm on Windows 7.

Hello , I have problems with visual c ++ 2011 to compile the CMDL to obj.

Build started 01/10/2015 13:21:27.
     1>Project "C:\Users\DJ Poizzon\Desktop\SD\Download\dkcr_cmdl_to_obj-master\dkcr_cmdl_to_obj-master\cmdl_parser\cmdl_parser.vcxproj" on node 2 (build target(s)).
       Project file contains ToolsVersion="12.0". This toolset may be unknown or missing, in which case you may be able to resolve this by installing the appropriate version of MSBuild, or the build may have been forced to a particular ToolsVersion for policy reasons. Treating the project as if it had ToolsVersion="4.0". For more information, please see http://go.microsoft.com/fwlink/?LinkId=291333.
     1>C:\Program Files\MSBuild\Microsoft.Cpp\v4.0\Platforms\Win32\Microsoft.Cpp.Win32.Targets(518,5): error MSB8008: Specified platform toolset (v120) is not installed or invalid. Please make sure that a supported PlatformToolset value is selected.
     1>Done Building Project "C:\Users\DJ Poizzon\Desktop\SD\Download\dkcr_cmdl_to_obj-master\dkcr_cmdl_to_obj-master\cmdl_parser\cmdl_parser.vcxproj" (build target(s)) -- FAILED.

Build FAILED.

Time Elapsed 00:00:01.70.

I'm on Windows 7.

Aruki

2015-10-01 08:38:21 am

Use 2012.

jackoalan

2015-10-04 07:48:09 pm

Finally got around to documenting ANIM in MP2/3..

Check out the new hub-article:
http://www.metroid2002.com/retromodding/wiki/ANIM_(File_Format%29

MrHostille

2015-10-05 05:07:29 am

Quote:

Use 2012

Okay, but my license is that version 10 is the same key for the key 12 ?

Aruki

2015-10-05 05:49:20 am

The community version of VS2012/2013 is free.