Metroid Prime hacking research

MrSpEeDrUn

Edit history:

MrSpEeDrUn: 2014-05-22 06:02:57 pm

MrSpEeDrUn: 2014-05-22 06:02:38 pm

MrSpEeDrUn: 2014-05-22 06:00:09 pm

MrSpEeDrUn: 2014-05-22 05:58:12 pm

2014-05-22 05:56:44 pm

MP2 also a bit MP1 speedrunning

You can edit every Item right? (would be a miracle, if Bombs and Morph ball where on the exact same spot , 1:100+1:100 that'd mean it would've been 1:200 i think so that's pretty unlikely to happen) Putting some effort into editing certain items would probably make a lot sence. (so it's much more unlikely to have those extremely annoying OoB's or even to have the game impossible to complete) That could be really amazing, since theres so many possibillities and it'd make the game quite different and refreshing. For example wave and plasma beam. having it on a bad spot completely ruins the game. Actually not sure about ice beam. (might be a good idea to make the player start with all 4 beams then)
Or at least wave, it just blocks way too much. Plasma and ice would actually be pretty unlikely to ruin it. (just like a 10% chance or so, kay, just a random guess but, thinking about how many items you can reach at the beginning)
Now, that i think about it, its actually pretty likely to finish the game, when the items are completely random, maybe like a 30% chance or so, so i think on your next run you might be able to do it already. But having the run screwed over randomly would be annoying (especially after wallcrawling for hours) Defenitly amazing idea you had here with the randomizer, indeed. aiwebs_004

You can edit every Item right? (would be a miracle, if Bombs and Morph ball where on the exact same spot , 1:100+1:100 that'd mean it would've been 1:200 i think so that's pretty unlikely to happen) Putting some effort into editing certain items would probably make a lot sence. (so it's much more unlikely to have those extremely annoying OoB's or even to have the game impossible to complete) That could be really amazing, since theres so many possibillities and it'd make the game quite different and refreshing. For example wave and plasma beam. having it on a bad spot completely ruins the game. Actually not sure about ice beam. (might be a good idea to make the player start with all 4 beams then)
Or at least wave, it just blocks way too much. Plasma and ice would actually be pretty unlikely to ruin it. (just like a 10% chance or so, kay, just a random guess but, thinking about how many items you can reach at the beginning)
Now, that i think about it, its actually pretty likely to finish the game, when the items are completely random, maybe like a 30% chance or so, so i think on your next run you might be able to do it already. But having the run screwed over randomly would be annoying (especially after wallcrawling for hours) Defenitly amazing idea you had here with the randomizer, indeed.[aiwebs_004]

Aruki

Edit history:

Aruki: 2014-05-22 06:26:15 pm

Aruki: 2014-05-22 06:25:26 pm

2014-05-22 06:24:38 pm

I don't think it would really be that hard to make it possible to complete the game every time. One way to do it that comes to mind is to set a list of requirements for each item spawn, and randomize each section of the game in batches; randomize all accessible locations, then check which new locations are accessible with the currently spawned items, and then randomize those locations. Then it's simply a matter of redoing the last batch if it ever hits a point where there are no more new accessible spawns.

TheMG2

2014-05-22 08:27:43 pm

Question: Do you want the randomizer to be completable, or do you want to theoretically be able to collect all the items? Because if the latter, there are some obvious scenarios where the latter can't happen: Omega Pirate has X-Ray Visor, Phazon Elite has main Power Bombs (I've always been told main power bombs is what allows you to activate phazon elite, it isn't getting the central dynamo item, right?).

MilesSMB

Edit history:

MilesSMB: 2014-05-22 08:54:59 pm

2014-05-22 08:43:19 pm

It's tied to the item in Central Dynamo.

Whether a room layer loads can't be tied to having a specific item in your inventory; room layers are turned on/off by a Special Function object in a specific room (for instance, Central Dynamo has several Special Functions that change room layers, that are activated when you collect the main PB). Now, you can essentially have it tied to a specific item by just moving the Special Function object to the room with the item (which I had to do for Artifacts) but it's not as simple as "you have main PBs so this layer is turned on".

About 100% completion, ideally the randomizer will have a lot of options. One of them would be whether the game is guaranteed to be completable 100% or not.

KentaKurodani

2014-05-23 08:27:46 am

Metroid 2 Ho! (and Zm soon to come!)

You guys really need to get in contact with another guy I know working on a Prime editor, antidote from the MetConst board. He hasn't been posting much, but he's been making fair progress on IRC

Hazel

Edit history:

Hazel: 2014-05-23 09:22:01 am

Hazel: 2014-05-23 09:21:48 am

Hazel: 2014-05-23 09:21:16 am

Hazel: 2014-05-23 09:20:31 am

2014-05-23 09:19:58 am

rocks, locks, and invisible blocks

Quote from MilesSMB:

I can't tell if Kirby is joking or not.

Quote from MrSpEeDrUn:

awesome, watched the recording, though couldn't you IS the item behind that tree super missile thing in chozo ruins?
I think you'd have enough energy for that.

I remember having the game crash when trying to IS in Main Plaza before, though that was a long time ago so who knows. Anyway there were several expansions I could've tried for, but every suit and Wave ended up being in the Mines so it was impossible to complete.

You can't collect that missile with IS in normal game-play, for some reason. Kind of like how you can't get the missile in the spider-ball-tower-thingy in Phendrana Shorelines. I'm pretty sure the other two missiles and energy tank can be collected all at the same time, though.

Antidote

2014-05-24 12:46:04 am

Paraxade: I know how to decompress MP2 and 3 MREA files, however I'm still a ways away from actually being able to render them, some details about the material section are holding me back from doing it properly, I'd greatly appreciate it we could help eachother out, I have a viewer that's able to load and view ALL CMDL files from the Primes, and all of MP1's Maps, and I've even got some support for DKCR.
If you're interested my github repo is here:
https://github.com/Antidote/MetPrimeTools

Aruki

Edit history:

Aruki: 2014-05-24 01:09:07 am

Aruki: 2014-05-24 01:07:44 am

2014-05-24 01:01:25 am

Yeah sure, helping each other out sounds fine with me. I've actually just been working on trying to get MREA files from Echoes decompressed/recompressed for Miles today. I'm getting stuck reading 0xC000 where the compressed block size should be, but aside from that I have it. Do you know anything about that?

edit: Also, I recall reading you were having trouble making custom PAKs? The only thing I can think of that might be screwing you up is that the MLVL file is required to be listed in the filename table. If it's not there, the game crashes. That's the only real catch as far as I know... the game doesn't even care whether files are compressed or not, you can leve everything uncompressed if you want (although it seems it doesn't like if you try to compress some formats). Aside from that just make sure all the data listed is accurate. I'm not really sure what's up with the duplicate files yet but I think they're there for optimization purposes; they aren't required, the game can load the pak fine without any duplicates.

Yeah sure, helping each other out sounds fine with me. I've actually just been working on trying to get MREA files from Echoes decompressed/recompressed for Miles today. I'm getting stuck reading 0xC000 where the compressed block size should be, but aside from that I have it. Do you know anything about that?

edit: Also, I recall reading you were having trouble making custom PAKs? The only thing I can think of that might be screwing you up is that the MLVL file is required to be listed in the filename table. If it's not there, the game crashes. That's the only real catch as far as I know... the game doesn't even care whether files are compressed or not, you can leve everything uncompressed if you want (although it seems it doesn't like if you try to compress some formats). Aside from that just make sure all the data listed is accurate. I'm not really sure what's up with the duplicate files yet but I think they're there for optimization purposes; they aren't required, the game can load the pak fine without any duplicates.

Antidote

Edit history:

Antidote: 2014-05-24 01:13:25 am

2014-05-24 01:12:21 am

I haven't attempted to create paks since then, but knowing what I know now it shouldn't be difficult at all to write a program to do so.

Also how MREA data is compressed varies widely from map to map, you can't always rely on it being at 0xC000

If you have a Skype account it'd be easier for us to communicate that way.

Aruki

2014-05-24 01:14:26 am

Uhh, I'm not relying on it being at 0xC000. I'm consistently reading across multiple files in different places the value 0xC000 in the place where it should be saying the size of the compressed block.

Yeah, paraxade on skype.

DJGrenola

2014-05-24 12:46:36 pm

how large is the size field? is it 16-bit?

might 0xc000 mean "use the same size as last block", or "use some default size X" ?

you've probably thought of those already though

Aruki

2014-05-24 04:11:42 pm

yeah, it's 16 bit. Doesn't seem like either of those unfortunately. I'm wondering if it's a bit flag, though. All the valid size values I've seen only use the lower 14 bits, while 0xC000 only uses the top two bits.

DJGrenola

2014-05-24 04:37:35 pm

not sure I have any other ideas except for setting up watchpoints in dolphin to see what the game does with those blocks

DJGrenola

Edit history:

DJGrenola: 2014-05-24 05:00:10 pm

2014-05-24 04:59:36 pm

I have skippy's annotated disassembly of the decompressor, the answer may be in there somewhere

DJGrenola

2014-05-24 04:59:56 pm

no idea whether skip is ok with me distributing it tho

Aruki

Edit history:

Aruki: 2014-05-24 05:04:53 pm

Aruki: 2014-05-24 05:03:39 pm

2014-05-24 05:03:17 pm

hmm, you could PM it to me maybe if you don't want to post it publicly? not sure I'd even be able to make any sense of it though

Could also have a look through it yourself and post back if you find anything relevant

DJGrenola

2014-05-24 05:12:22 pm

ok I'll dig it out later

Aruki

Edit history:

Aruki: 2014-05-25 04:23:32 am

Aruki: 2014-05-25 02:28:11 am

Aruki: 2014-05-25 12:49:59 am

Aruki: 2014-05-25 12:49:41 am

Aruki: 2014-05-25 12:07:55 am

Aruki: 2014-05-24 05:28:44 pm

Aruki: 2014-05-24 05:24:04 pm

2014-05-24 05:22:57 pm

Also may as well post what I know about the compression so far since I haven't explained it in detail yet. The way the MP2 MREAs work is, basically the entire file after the header is compressed. The header contains a table of section sizes starting at 0x80, similar to Prime 1 (sections are typically organized groups of data - for instance the first section is always material data, there's a vertex coordinate section on every mesh, etc). Following that is a list of compressed clusters with four values each. First is unknown (seems to always be 0x120 higher than the second value), the second is the size of the cluster decompressed, the third is the size of the cluster compressed, and the fourth is how many "regular" sections are inside that cluster. Occasionally there are uncompressed 32-byte-large clusters listed as well.

Normally in Metroid Prime files at the end of a chunk of data, they pad the file until the next multiple-of-32 offset. On compressed sections, they pad the beginning instead of the end. Following that is where the compressed data begins. The LZO-compressed data is made up of multiple blocks of compressed data that needs to be decompressed separately. Each block starts with a 16-bit size value for the current block. So you read the size value, then read that many bytes following that, and pass those to the decompressor. That'll usually put you right after a 0000 (which typically marks the end of a compressed block) and right before another size value.

The issue I'm talking about is, in a lot of files there's a value of 0xC000 listed where there should be a size value. No idea what it means yet, except that trying to find actually valid size values nearby has not worked out so far, even by trying to jump to the next 0000. On the odd occasion I find a file that doesn't have any C000s though, I can decompress it perfectly.

edit: WELP apparently it's stupidly obvious. C000 sections are not compressed and they are always 0x4000 bytes long. That's all there is to it. Wow.

edit 2: Alright, so it's a bit more complicated than that, but still simple. Each block is 0x4000 long at the most. Any size value listed as 0x4000 or lower is a compressed section, where the value is the size of the block. Any size value listed as 0xC000 or higher is an uncompressed section, where the value is subtracted from 0x10000 to get the size of the block.

Format seems to be totally cracked now, I'm fully decompressing every MREA I try with no issue now.

edit 3: As a side note, the compression used by paks on CMDLs/TXTRs/etc don't have C000 sections. All sizes in those files are absolute. They can also be higher than 0x4000.

Also may as well post what I know about the compression so far since I haven't explained it in detail yet. The way the MP2 MREAs work is, basically the entire file after the header is compressed. The header contains a table of section sizes starting at 0x80, similar to Prime 1 (sections are typically organized groups of data - for instance the first section is always material data, there's a vertex coordinate section on every mesh, etc). Following that is a list of compressed clusters with four values each. First is unknown (seems to always be 0x120 higher than the second value), the second is the size of the cluster decompressed, the third is the size of the cluster compressed, and the fourth is how many "regular" sections are inside that cluster. Occasionally there are uncompressed 32-byte-large clusters listed as well.

Normally in Metroid Prime files at the end of a chunk of data, they pad the file until the next multiple-of-32 offset. On compressed sections, they pad the beginning instead of the end. Following that is where the compressed data begins. The LZO-compressed data is made up of multiple blocks of compressed data that needs to be decompressed separately. Each block starts with a 16-bit size value for the current block. So you read the size value, then read that many bytes following that, and pass those to the decompressor. That'll usually put you right after a 0000 (which typically marks the end of a compressed block) and right before another size value.

The issue I'm talking about is, in a lot of files there's a value of 0xC000 listed where there should be a size value. No idea what it means yet, except that trying to find actually valid size values nearby has not worked out so far, even by trying to jump to the next 0000. On the odd occasion I find a file that doesn't have any C000s though, I can decompress it perfectly.

edit: WELP apparently it's stupidly obvious. C000 sections are not compressed and they are always 0x4000 bytes long. That's all there is to it. Wow.

edit 2: Alright, so it's a bit more complicated than that, but still simple. Each block is 0x4000 long at the most. Any size value listed as 0x4000 or lower is a compressed section, where the value is the size of the block. Any size value listed as 0xC000 or higher is an uncompressed section, where the value is subtracted from 0x10000 to get the size of the block.

Format seems to be totally cracked now, I'm fully decompressing every MREA I try with no issue now.

edit 3: As a side note, the compression used by paks on CMDLs/TXTRs/etc don't have C000 sections. All sizes in those files are absolute. They can also be higher than 0x4000.

TheMG2

2014-05-25 12:52:01 am

Oh man. This is going to be good.

Antidote

Edit history:

Antidote: 2014-05-25 11:01:17 am

2014-05-25 06:51:08 am

Fortunately for our sanity DKCR has switched back to zlib and that nonsense is non-existent, all we have to do is check for 0x78DA and boom we know it's compressed.

EDIT:
Literally as of 10 minutes ago: