Metroid Prime hacking research

2015-03-25 01:48:49 pm

what I would do to get my hands on one of these models in blender...

2015-03-25 02:27:38 pm

Quote from wowsers:

what I would do to get my hands on one of these models in blender...

I will eventually be releasing a utility to extract an entire .gcm of MP to .blend files. It's too rough to make public ATM.

Aruki

Edit history:

Aruki: 2015-03-27 03:11:16 am

Aruki: 2015-03-27 03:10:37 am

Aruki: 2015-03-26 11:06:05 pm

2015-03-26 11:05:00 pm

Did some work today with the FONT format. I wanted to try out rendering text using the game's fonts using the info on the wiki that DJGrenola and jackoalan came up with.

(This is Corruption's Deface font; NoARAM/7d69abb59fa59761.FONT)

There are four different versions of the FONT format that we know of, but as it turns out they're all rather similar to each other and Retro was just really liberal with the version number for this format. Version 1 (MP1 kiosk demo) is identical to version 2 (MP1 0-00), except for the addition of one value. In version 4 (MP1 PAL, MP2), they started doing something interesting with the font textures - they encode multiple glyphs in the same spot on the texture by putting them on different RGBA channels. Version 4 has an added value to track which channel the glyph is on. A bunch of values were also reduced from 32 bits down to 8 or 16 bits. Aside from that, it's the same layout. As far as I can tell, version 5 (MP3/DKCR) is identical to version 4, except the texture ID is 64 bits now.

All this info + more is going to be on the wiki soon... need to update the page with some new info. There might be some other differences; it's hard to say since there's still a few unknowns in the header.

Once I have the kinks worked out I'm probably going to use these to display info in the viewport. I haven't added kerning yet, but the way the format is set up should make that really easy to implement.

Did some work today with the FONT format. I wanted to try out rendering text using the game's fonts using the info on the wiki that DJGrenola and jackoalan came up with.

[img]http://i.imgur.com/MpXGLXO.png[/img]
(This is Corruption's Deface font; NoARAM/7d69abb59fa59761.FONT)

There are four different versions of the FONT format that we know of, but as it turns out they're all rather similar to each other and Retro was just really liberal with the version number for this format. Version 1 (MP1 kiosk demo) is identical to version 2 (MP1 0-00), except for the addition of one value. In version 4 (MP1 PAL, MP2), they started doing something interesting with the font textures - they encode multiple glyphs in the same spot on the texture by putting them on different RGBA channels. Version 4 has an added value to track which channel the glyph is on. A bunch of values were also reduced from 32 bits down to 8 or 16 bits. Aside from that, it's the same layout. As far as I can tell, version 5 (MP3/DKCR) is identical to version 4, except the texture ID is 64 bits now.

All this info + more is going to be on the wiki soon... need to update the page with some new info. There might be some other differences; it's hard to say since there's still a few unknowns in the header.

Once I have the kinks worked out I'm probably going to use these to display info in the viewport. I haven't added kerning yet, but the way the format is set up should make that really easy to implement.

[img]http://i.imgur.com/76kLa6p.png[/img]

DJGrenola

Edit history:

DJGrenola: 2015-03-27 08:26:20 pm

2015-03-27 08:19:30 pm

ah that's interesting about the channels.

the thing I wanted to look into with this (but thus far couldn't be bothered) was writing something that would use libfreetype to take a TrueType font file as input, and then magically spit out a texture and FONT file that could be loaded into the game. but there's a bit of work involved and I'm lazy. so that's a project for someone if they want it.

Aruki

Edit history:

Aruki: 2015-03-28 01:03:26 am

2015-03-28 12:51:56 am

yep. It might be that they actually modify the palette indices themselves and then load each layer as a separate texture... this seems feasible since Dolphin dumps each layer separately, but it's hard to say for sure without looking at the code. Utilizing the RGBA channels for this still works perfectly fine, even if it's not what Retro did, since there's never more than 4 glyph layers. DKCR starts using 8-bit palette indices for font textures, so they could theoretically embed up to 8 layers per texture (in which case the RGBA theory would be dead), but they still never go over 4. Either way Retro's definitely not using the texture data as-is.

The format supports outlines using the different channels/layers as well. I should be able to support that since I just found the value that indicates the texture format. Version 1/2 support outlines as well, btw, so I guess they started doing the multiple layer thing a bit earlier than I initially thought. Still doesn't support multiple glyph layers though, since the glyphs don't have any value to track which layer they're on.

yep. It might be that they actually modify the palette indices themselves and then load each layer as a separate texture... this seems feasible since Dolphin dumps each layer separately, but it's hard to say for sure without looking at the code. Utilizing the RGBA channels for this still works perfectly fine, even if it's not what Retro did, since there's never more than 4 glyph layers. DKCR starts using 8-bit palette indices for font textures, so they could theoretically embed up to 8 layers per texture (in which case the RGBA theory would be dead), but they still never go over 4. Either way Retro's definitely not using the texture data as-is.

The format supports outlines using the different channels/layers as well. I should be able to support that since I just found the value that indicates the texture format. Version 1/2 support outlines as well, btw, so I guess they started doing the multiple layer thing a bit earlier than I initially thought. Still doesn't support multiple glyph layers though, since the glyphs don't have any value to track which layer they're on.

jackoalan

Edit history:

jackoalan: 2015-03-28 01:40:09 am

2015-03-28 01:39:59 am

ANIM resources were rather tricky to figure out... Here's what happens when the data is (incorrectly) applied using Euler Angles:

wowsers

2015-03-28 09:43:06 am

"oh my" indeed... I wish I could actually contribute, but the only experience I have with anything here is the lazy encryption I made for my own game-engine. It just adds 4 bits to the beginning and end of the text string making it unreadable...

Antidote

2015-03-28 11:51:49 am

Hey, she has urges too!

Antidote

2015-03-28 07:02:53 pm

Metroid Prime 3 UV Animations in Retro View

Aruki

Edit history:

Aruki: 2015-03-30 01:38:10 pm

2015-03-30 01:35:00 pm

Minor interesting point: Miles and I worked out how the script loader functions work in the demo/final game. Thanks to that we found the names of some unused objects. Unfortunately a bunch of them have been removed from the final, but here's a list of the unused slots that had valid objects attached to them in the demo:

* 0x1: Enemy - removed in 0-00; load function immediately returns 0
* 0x12: JumpPoint - removed in 0-00; load function immediately returns 0
* 0x22: MapStation - removed in 0-00; load function immediately returns 0. Map Stations are done using a SpecialFunction in the final game.
* 0x32: DebugCameraWaypoint - still exists!
* 0x38: DockAreaChange - still exists!
* 0x3C: Sova - COMPLETELY gone in 0-00; no load function gets attached to this slot at all
* 0x41: AIKeyframe - still exists!
* 0x52: WorldTeleporter - it is the same WorldTeleporter as slot 0x62, not an earlier version. Same load function. For some reason it gets two slots.
* 0x59: SaveStation - removed in 0-00; load function immediately returns 0. Like MapStation, this is done using a SpecialFunction in the final game.
* 0x65: FogVolume - still exists!

I'm pretty disappointed Sova is gone. I guess if we want to see what that object does we'll need to experiment on the kiosk demo instead of the final.

Minor interesting point: Miles and I worked out how the script loader functions work in the demo/final game. Thanks to that we found the names of some unused objects. Unfortunately a bunch of them have been removed from the final, but here's a list of the unused slots that had valid objects attached to them in the demo:

* 0x1: Enemy - removed in 0-00; load function immediately returns 0
* 0x12: JumpPoint - removed in 0-00; load function immediately returns 0
* 0x22: MapStation - removed in 0-00; load function immediately returns 0. Map Stations are done using a SpecialFunction in the final game.
* 0x32: DebugCameraWaypoint - still exists!
* 0x38: DockAreaChange - still exists!
* 0x3C: Sova - COMPLETELY gone in 0-00; no load function gets attached to this slot at all
* 0x41: AIKeyframe - still exists!
* 0x52: WorldTeleporter - it is the same WorldTeleporter as slot 0x62, not an earlier version. Same load function. For some reason it gets two slots.
* 0x59: SaveStation - removed in 0-00; load function immediately returns 0. Like MapStation, this is done using a SpecialFunction in the final game.
* 0x65: FogVolume - still exists!

I'm pretty disappointed Sova is gone. I guess if we want to see what that object does we'll need to experiment on the kiosk demo instead of the final.

MilesSMB

2015-03-30 03:58:45 pm

There's also one more unused object (0x78) that still exists and isn't in the demo; appears to be an enemy. No clue what it could be.

jackoalan

2015-03-30 04:47:53 pm

Having all ANIMs available to view in Blender makes it easy to compare revisions of the game's data. Most notably, the changed Phazon Suit acquire animation:

Old Version (GM8E01-0-00):

New Version (R3ME01-0-00):

jackoalan

2015-03-31 10:19:57 pm

Ridley was clearly a devious bastard to animate.. 89 bones!!

wowsers

2015-04-01 02:37:56 pm

are there breakpoints in the curves? if not, did they have breakpoints in the game or was it just all smooth curves?

Aruki

2015-04-01 02:45:23 pm

Working on PWE's model/material editor. Here's a sneak peek at how the UI is looking currently.

jackoalan

Edit history:

jackoalan: 2015-04-01 05:16:05 pm

2015-04-01 05:15:51 pm

Quote from wowsers:

are there breakpoints in the curves? if not, did they have breakpoints in the game or was it just all smooth curves?

In blender-animation terminology, keys are used to shape out the fcurves you see (with little handles you can drag around and apply béziers to).. I have the keys hidden in this screenshot; It's hard to see the curves with them on. Blender will attempt to geometrically fit a smooth-curve in the graph view, as a tool for animators to detect and fix 'jitters'.. In reality, keys are placed at 30-fps in this particular animation, with the game-engine interpolating them up to 60-fps.. I've gotten blender to do the same thing.

I'm sure Derek Bonikowski and his animation crew used an auto-keying system in 3ds-max; directly posing Ridley's bones by manipulating them in-scene, generating the curves on-the-fly.. IK (Inverse Kinematics) rigs may have also lent some animation-assistance.

Aruki

2015-04-03 12:54:25 am

So I have a functional UV anim-editing interface now, with real-time previews, which is pretty damn cool. Made a video to show a few people earlier, so hey, why not upload it?

Keep in mind everything you see here is a WIP and most of it isn't even functional yet.

jackoalan

Edit history:

jackoalan: 2015-04-04 06:08:21 am

2015-04-04 06:06:26 am

Another CodeWarrior episode:

Aruki

Edit history:

Aruki: 2015-04-04 05:06:30 pm

Aruki: 2015-04-04 05:06:08 pm

Aruki: 2015-04-04 05:06:07 pm

Aruki: 2015-04-04 05:05:45 pm

2015-04-04 04:59:18 pm

^ basically what this is is a really really confusing way of checking whether the bits 0x100 or 0x40 are set on a value. CodeWarrior loves to make our lives harder.

Code:

The PPC assembly code:

andi. r3, r19, 0x140
neg r0, r3
or r0, r0, r3
srwi. r25, r0, 31

Translates to:

r3 = r19 & 0x140; // r3 now contains the value of two bits from r19 - 0x100 and 0x40
r0 = -r3;         // r0 now contains the negated form of r3
r0 |= r3;         // r0 now contains every bit that's set in either r0 or r3
r25 = r0 >> 31;   // r25 now contains the value of r0 shifted right 31 bits

What this accomplishes: The output is in r25 - a one-bit value that will be 0 if r3 was initialized to 0 in the first line, and 1 if it was initialized to any other number.
Thus, this is a very confusing means of checking whether either bit 0x100 or 0x40 is set on the input value (r19).

Aruki

Edit history:

Aruki: 2015-04-05 02:31:06 am

Aruki: 2015-04-05 02:28:19 am

2015-04-05 02:27:29 am

On a side note, yay MP1 materials are near 100% cracked! Trying to get my material cooker (for resaving materials in the game's format) working 100% so I looked into some of the remaining unknown values a little further. I also figured out some of the unknowns on the geometry data... turns out the surface headers (submeshes, but apparently Retro called them surfaces) contain some info that's used to render the reflections. Also, found an interesting little feature that the game never actually uses - terrain meshes can have their visibility turned on and off on a per-visor basis. So you could have a terrain mesh that's only visible with X-Ray, with Thermal, etc. The same value also allows you to set whether a terrain mesh appears hot or cool in the thermal visor. Been wondering how stuff like that was done for a while so it's nice to finally know.

The only unknown left on MP1 is the value immediately following the vertex description flags. I have no idea what it is honestly, and I've spent a lot of time trying to figure it out. The game does access it, but it's a somewhat confusing section of the material code that involves a lot of weird global variables and I honestly can't tell whether what it's doing with that value actually does anything. At any rate, changing it doesn't appear to cause any changes ingame, yet I seem to remember having the game crash from messing around with it. So I really have no idea.

Echoes introduced some changes to the material format - two new values, a position matrix index/texture matrix indices can be specified per-vertex now, and there's some different flag settings. I'm gonna try looking into how those work a little bit before moving on, although that might end up being difficult because Echoes doesn't conveniently have map files lying around like MP1 does (or its demo at least). Echeos DOES have a .sel file with some symbols in Trilogy, but I'm not sure how well that matches up with the Gamecube version - the build date for the Trilogy version is mid-2008.

On a side note, [url=http://www.metroid2002.com/retromodding/wiki/Materials_(Metroid_Prime)]yay MP1 materials are near 100% cracked![/url] Trying to get my material cooker (for resaving materials in the game's format) working 100% so I looked into some of the remaining unknown values a little further. I also figured out some of the unknowns on the geometry data... turns out the surface headers (submeshes, but apparently Retro called them surfaces) contain some info that's used to render the reflections. Also, found an interesting little feature that the game never actually uses - terrain meshes can have their visibility turned on and off on a per-visor basis. So you could have a terrain mesh that's only visible with X-Ray, with Thermal, etc. The same value also allows you to set whether a terrain mesh appears hot or cool in the thermal visor. Been wondering how stuff like that was done for a while so it's nice to finally know.

The only unknown left on MP1 is the value immediately following the vertex description flags. I have no idea what it is honestly, and I've spent a lot of time trying to figure it out. The game does access it, but it's a somewhat confusing section of the material code that involves a lot of weird global variables and I honestly can't tell whether what it's doing with that value actually does anything. At any rate, changing it doesn't appear to cause any changes ingame, yet I seem to remember having the game crash from messing around with it. So I really have no idea.

Echoes introduced some changes to the material format - two new values, a position matrix index/texture matrix indices can be specified per-vertex now, and there's some different flag settings. I'm gonna try looking into how those work a little bit before moving on, although that might end up being difficult because Echoes doesn't conveniently have map files lying around like MP1 does (or its demo at least). Echeos DOES have a .sel file with some symbols in Trilogy, but I'm not sure how well that matches up with the Gamecube version - the build date for the Trilogy version is mid-2008.

Antidote

Edit history:

Antidote: 2015-04-05 03:55:04 am

2015-04-05 03:54:47 am

With a bit of effort, Jackoalan was able to get RetroView working on OS X, so RetroView is now supported on linux, windows AND OS X! Expect an official beta release of RV in the near future.

jackoalan

2015-04-05 04:43:33 am

^ Screenshot Evidence:

Baby Sheegoth

2015-04-05 11:13:33 am

Oh man that is so sweet

MilesSMB

Edit history:

MilesSMB: 2015-04-10 05:24:41 pm

MilesSMB: 2015-04-10 04:51:47 pm

MilesSMB: 2015-04-10 04:46:25 pm

MilesSMB: 2015-04-10 04:46:23 pm

MilesSMB: 2015-04-10 02:08:03 pm

MilesSMB: 2015-04-10 01:45:53 pm

2015-04-06 10:27:01 pm

Quote from MilesSMB:

There's also one more unused object (0x78) that still exists and isn't in the demo; appears to be an enemy. No clue what it could be.

Turns out it's probably "IceZoomer". Weird.

On a related note, I found 14 unused objects referenced in Echoes. Echoes SCLYs use a four char ID for each object type instead of just a number, which is much more helpful for trying to figure out what they are. Here's the IDs and my guesses:

CHOG - ChozoGhost
COIN - Coin...
DBAR - DestructibleBarrier
DBR1 - Debris (Prime 1 has two types of debris, Echoes only uses the second type)
DTRO - DamageableTriggerOrientated (TriggerOrientated is used, this is presumably the same thing except damageable)
EMPU - EMPulse
MIDI - Midi
PARA - Parasite
PSCH - PlayerStateChange
RIPL - Ripple
RIPR - Ripper
RSFA - RsfAudio
SILH - Silhouette
SPNK - SpankWeed

Aruki

Edit history:

Aruki: 2015-04-08 02:06:43 am

2015-04-07 07:31:26 pm

Just figured out the last unknown on MP1 materials. It assigns materials that have the same settings, but with different textures, into groups. This way the game can internally track which material group is currently set. Then after loading textures, it can check whether the group the current material is in is already set up, and if so, skip running through the rest of the material setup. This certainly explains a lot - its placement in the material data, the reason why changing it seems to not do anything sometimes, the reason why changing it crashes the game sometimes (proper vertex description not being set so the GPU can't read the model display lists properly), and the reason why it prevented my custom animations from running ingame an hour ago.

MP1 materials are 100% cracked now! *confetti* Not much luck on the new Echoes stuff, though.

EDIT: I'm now able to make changes in my material editor, resave the model, and have the changes show up ingame. Mainly just gotta make the rest of the UI functional now!

Just figured out the last unknown on MP1 materials. It assigns materials that have the same settings, but with different textures, into groups. This way the game can internally track which material group is currently set. Then after loading textures, it can check whether the group the current material is in is already set up, and if so, skip running through the rest of the material setup. This certainly explains a lot - its placement in the material data, the reason why changing it seems to not do anything sometimes, the reason why changing it crashes the game sometimes (proper vertex description not being set so the GPU can't read the model display lists properly), and the reason why it prevented my custom animations from running ingame an hour ago.

MP1 materials are 100% cracked now! *confetti* Not much luck on the new Echoes stuff, though.

EDIT: I'm now able to make changes in my material editor, resave the model, and have the changes show up ingame. Mainly just gotta make the rest of the UI functional now!