Metroid Prime hacking research

Aruki

2015-04-16 07:03:10 pm

What do the colors indicate?

jackoalan

Edit history:

jackoalan: 2015-04-16 07:31:55 pm

2015-04-16 07:30:21 pm

there is one color selected from a palette of 8 indicating each dimensional octant.

The code to generate the colors was purely temporary, and I've taken it down.
My findings from these colors are on the wiki article.

Nisto

Edit history:

Nisto: 2015-04-17 10:55:41 am

Nisto: 2015-04-17 10:55:39 am

Nisto: 2015-04-17 09:53:36 am

Nisto: 2015-04-17 09:52:57 am

Nisto: 2015-04-17 09:44:17 am

Nisto: 2015-04-17 09:38:43 am

Nisto: 2015-04-17 09:12:08 am

Nisto: 2015-04-17 09:10:29 am

2015-04-17 09:09:27 am

I sent a PM to Parax (on VGMdb) a couple of days ago regarding the Sample Directory format (stored in AGSC files), but it seems he may have stopped visiting the site. So I just came here to tell you that there's a couple of things to update on your Wiki for the Sample Directory tables:

- Table B: lyn2 ("Loop context sample history 2") is actually stored before lyn1 ("Loop context sample history 1"). I verified this by batch-converting a bunch of audio files with Nintendo's official DSPADPCM encoder (from the Dolphin SDK) and comparing the DSP header values to those stored in the Sample Directory structure.

- Table A: I don't know if you were aware, but since loops are expressed in raw samples, rather than nibbles (as opposed to a standard DSP header), you need to convert to nibbles:

Code:

int32_t dsp_samples_to_nibbles(int32_t samples) {
    int32_t whole_frames = samples/14;
    int32_t remainder = samples%14;

    if (remainder>0)
        return (whole_frames*16) + remainder + 2;
    else
        return whole_frames*16;
}

I hope it's of help.

I sent a PM to Parax (on VGMdb) a couple of days ago regarding the Sample Directory format (stored in AGSC files), but it seems he may have stopped visiting the site. So I just came here to tell you that there's a couple of things to update on your Wiki for the Sample Directory tables:

- Table B: lyn2 ("Loop context sample history 2") is actually stored before lyn1 ("Loop context sample history 1"). I verified this by batch-converting a bunch of audio files with Nintendo's official DSPADPCM encoder (from the Dolphin SDK) and comparing the DSP header values to those stored in the Sample Directory structure.

- Table A: I don't know if you were aware, but since loops are expressed in raw samples, rather than nibbles (as opposed to a standard DSP header), you need to convert to nibbles:

[code]int32_t dsp_samples_to_nibbles(int32_t samples) {
    int32_t whole_frames = samples/14;
    int32_t remainder = samples%14;

if (remainder>0)
        return (whole_frames*16) + remainder + 2;
    else
        return whole_frames*16;
}[/code]

I hope it's of help.

Aruki

Edit history:

Aruki: 2015-04-17 01:01:39 pm

Aruki: 2015-04-17 12:56:08 pm

2015-04-17 11:17:58 am

Yeah, the wiki says it's in samples. We've had loop playback working correctly in decoded audio. I'll update the sample history thing, thanks.

edit: btw, this works too:

Code:

u32 NumSampleBytes = SampleIndex / 2;
u32 NumHeaderBytes = 1 + (SampleIndex / 14);
return NumSampleBytes + NumHeaderBytes;

jackoalan

Edit history:

jackoalan: 2015-04-17 08:14:01 pm

2015-04-17 08:05:53 pm

AROT is starting to come together in RetroView!!

This is a test showing how it functions..
RetroView is limited to rendering 20 meshes furthest from the camera, using simple conditional-recursion, no sorting required!

When the limit is removed, the rendering occurs in back-to-front order

Antidote

2015-04-17 08:14:16 pm

Nice, hard to believe that only a few days ago we knew virtually nothing about AROT.

jackoalan

2015-04-18 04:15:24 am

At long last! AROT is serving its purpose hand-in-hand with planar frustum-culling to speed up RetroView!!
On the road to consistent 60-fps!!

In this video, the view is presented in the normal 55-degree field-of-view, but in the center of the viewport is a virtual 20-degree field-of-view for culling..

Antidote

2015-04-22 03:23:54 am

I started experimenting with ActorRotate and got some near perfect results!

MrSinistar

2015-04-22 03:55:58 am

I've been messing around with the SCLY Sound Class objects and figured out the majority of the booleans. Doing sound mods are now pretty close to being a real thing! http://www.metroid2002.com/retromodding/wiki/User:MrSinistar/Sandbox#SCLY_Audio_Object

Antidote

Edit history:

Antidote: 2015-04-23 10:34:21 am

2015-04-23 10:34:01 am

Spent a good portion of last night scouring the internet for information about FSB, and wouldn't you know that FMOD's forums turned out to be the best gold mine ever?! So FSB conversion is now possible! :D

puelo

Edit history:

puelo: 2015-04-26 04:45:46 pm

puelo: 2015-04-26 04:43:21 pm

2015-04-26 04:42:40 pm

Is there something similar going on for DKCR? I stumbled over the work of Parax on the DKCR models and the different File Format on the wiki page here. I also tried my luck and programmed a small (VERY HACKY) tool to convert .cmdl files to .obj just to view them (.obj is not really good for anything further).

My tool is largely build on the research of Parax (and others who worked on the information on the wiki page). What gave me a few headaches was the fact that the vertex attribute flags can contain bits where i would need to skip certain bytes in the submeshes. This was already known through the MaxScript Parax wrote. The difference to MP is that it can either be 1 or 2 bytes (at first i only skipped the one byte). This should maybe get added to the wikipage. I could try to write up the Geometry Page if there is demand.

Code:

if ((vertexAttributesFlags[matID] & 0xFF000000) == 0x1000000) bytesToSkip = 1;	// Don't know if this is completely true yet, but this happens from time to time...
if ((vertexAttributesFlags[matID] & 0xFF000000) == 0x3000000) bytesToSkip = 2;

I only found out like 2 hours ago that Parax also wrote a MaxScript for the CMDL Format of DKCR. One thing which i could adapt from it was the weird way one would need to calculate the vertex positions if they are stored as shorts. Any idea why this happens? Originally i had them just divided by 0x2000 to get them in range of -4 and 4. But it seems like the method Parax is doing is the right one (at least it works :P).

Source Code can be found here: https://github.com/KammererTob/dkcr_cmdl_to_obj
(Be advised that the code is very very messy and surely still full of bugs. So use at own "risk" :P)

SO. Since this is a metroid prime board i was wondering if there is a similar place for the research on DKCR (if any is done).
Sorry if this is off-topic.

Is there something similar going on for DKCR? I stumbled over the work of Parax on the DKCR models and the different File Format on the wiki page here. I also tried my luck and programmed a small (VERY HACKY) tool to convert .cmdl files to .obj just to view them (.obj is not really good for anything further).
[img]http://i.imgur.com/7MndgJi.png[/img]

My tool is largely build on the research of Parax (and others who worked on the information on the wiki page). What gave me a few headaches was the fact that the vertex attribute flags can contain bits where i would need to skip certain bytes in the submeshes. This was already known through the MaxScript Parax wrote. The difference to MP is that it can either be 1 or 2 bytes (at first i only skipped the one byte). This should maybe get added to the wikipage. I could try to write up the Geometry Page if there is demand.
[code]
if ((vertexAttributesFlags[matID] & 0xFF000000) == 0x1000000) bytesToSkip = 1;	// Don't know if this is completely true yet, but this happens from time to time...
if ((vertexAttributesFlags[matID] & 0xFF000000) == 0x3000000) bytesToSkip = 2; 
[/code]

I only found out like 2 hours ago that Parax also wrote a MaxScript for the CMDL Format of DKCR. One thing which i could adapt from it was the weird way one would need to calculate the vertex positions if they are stored as shorts. Any idea why this happens? Originally i had them just divided by 0x2000 to get them in range of -4 and 4. But it seems like the method Parax is doing is the right one (at least it works :P).

Source Code can be found here: https://github.com/KammererTob/dkcr_cmdl_to_obj
(Be advised that the code is very very messy and surely still full of bugs. So use at own "risk" :P)

SO. Since this is a metroid prime board i was wondering if there is a similar place for the research on DKCR (if any is done).
Sorry if this is off-topic.

Aruki

Edit history:

Aruki: 2015-04-26 04:57:25 pm

2015-04-26 04:53:20 pm

There isn't really another place for DKCR discussion that I know of... we tend to just talk about everything in here because DKCR shares a ton of similarities to Metroid Prime 3 anyway.

So, the thing with dividing the shorts - I have no idea why that's necessary. I believe they're just simply signed shorts and it seems like Max horribly screws them up if I just read them as signed, so I end up reading them as unsigned and then manually calculating where the signed value should be. It shouldn't really be necessary, but whatever.

For the vertex attribute flags, it's actually a flag of 8 bits - each bit indicates a matrix attribute on the vertex. This is notated on the materials page for Prime 1, but I'm not sure if it's on the Prime 3 page; I believe 0x1 is position matrix, and the rest are tex0 through tex6. It's used for skinning, apparently - I'm not 100% sure on how it works. (those settings are probably the same in DKCR but I haven't verified that)

Go ahead and update the Wiki with whatever information you found; it's a wiki for a reason :P Just gotta make an account first.

There isn't really another place for DKCR discussion that I know of... we tend to just talk about everything in here because DKCR shares a ton of similarities to Metroid Prime 3 anyway.

So, the thing with dividing the shorts - I have no idea why that's necessary. I believe they're just simply signed shorts and it seems like Max horribly screws them up if I just read them as signed, so I end up reading them as unsigned and then manually calculating where the signed value should be. It shouldn't really be necessary, but whatever.

For the vertex attribute flags, it's actually a flag of 8 bits - each bit indicates a matrix attribute on the vertex. This is notated on the materials page for Prime 1, but I'm not sure if it's on the Prime 3 page; I believe 0x1 is position matrix, and the rest are tex0 through tex6. It's used for skinning, apparently - I'm not 100% sure on how it works. (those settings are probably the same in DKCR but I haven't verified that)

Go ahead and update the Wiki with whatever information you found; it's a wiki for a reason :P Just gotta make an account first.

jackoalan

2015-04-26 04:58:01 pm

That reader code is actually not too shabby...
Antidote should probably chime in here, he has a lot of experience parsing DKCR CMDLs (including some Material interpretation)

The positions and normals are indeed stored as normalized, signed shorts:
https://github.com/RetroView/MetPrimeTools/blob/master/RetroView/src/io/src/CModelReader.cpp#L236

The Wii would use GXSetVtxAttrFmt to have the graphics-processor interpret them as fixed-point fractional values.. this would require a u8 'frac' value somewhere in the headers not currently known.

Aruki

2015-04-26 05:00:25 pm

Actually, the vertex format is set by the bottom 3 bits in the primitive type flag. It just sets the vertex format index; the actual formats themselves I'm pretty sure are hardcoded.

jackoalan

2015-04-26 05:01:02 pm

ahh, so it's a question of finding the constant value passed to 'frac' then

Aruki

Edit history:

Aruki: 2015-04-26 05:05:07 pm

2015-04-26 05:03:52 pm

Yeah, but it's not really hard to figure out by examining the data. Dividing by 0x2000 (putting the range from -4 to 4) seems to be correct because that's what makes the UV coordinates come out correctly. It's not a stretch to think positions are probably the same, but that should be easily verifiable because using the wrong value would introduce a scaling issue, and there are a lot of models that just straight up use floats instead of shorts, which should make a comparison pretty simple. That would correspond to a frac setting of 13, I think, if I'm understanding the function right.

jackoalan

2015-04-26 05:07:26 pm

cool!! good to know with the UV test.. the proof is in the pudding

puelo

Edit history:

puelo: 2015-04-26 05:10:25 pm

2015-04-26 05:09:23 pm

Might be a dumb question: But how to you debug or disassemble Wii Roms?

jackoalan

2015-04-26 05:10:13 pm

it's all standard PPC-EABI.. so if you're used to x86, you'll need to retrain yourself on PPC as well..

read up on RISC.. it'll help a lot

Aruki

Edit history:

Aruki: 2015-04-26 05:11:29 pm

2015-04-26 05:10:42 pm

If you have IDA, there's a loader for .dol files somewhere that makes it pretty simple to disassemble them. Also, DKCR has a .sel file containing symbols that you'll want to try to get imported... it's the least comprehensive symbol data we have out of every Retro game but it's better than nothing.

puelo

2015-04-26 05:11:27 pm

Meh. Kinda asked the wrong question, but your answers should still be helpful. Thank you!

jackoalan

2015-04-26 05:14:24 pm

basically, get comfortable with IDA and Dolphin's debugger (and really understand how they work).. static analysis (via IDA) combined with dynamic analysis (via dolphin) is a powerful one-two punch

Aruki

2015-04-26 05:15:18 pm

Oh yeah, the Dolphin debugger is stupidly helpful too.. that lets you set breakpoints in the code and view what values are in each register at any given moment.

jackoalan

2015-04-26 05:18:28 pm

i've also written an IDC script for IDA that exports your discovered symbols to a dolphin-compatible .map file:
https://gist.github.com/jackoalan/9a3e7e0c71f531686900

puelo

2015-04-26 05:27:26 pm

Wow nice stuff! Did not know that Dolphin could also be used for stuff like this