Metroid Prime hacking research

Edit history:

Aruki: 2015-01-19 05:18:15 pm

2015-01-19 05:12:45 pm

PakTool doesn't use minilzo, since it doesn't support compressing with lzo1x-999. It is possible I did something wrong using the full lzo library though I guess?

Also, PakTool does a lot of things poorly since it was my first attempt at writing a serious C++ program.. ever, basically. I think the new one will probably perform a fair bit better.

DJGrenola

Edit history:

DJGrenola: 2015-01-19 05:19:19 pm

2015-01-19 05:18:34 pm

I dunno, paktool doesn't ship with a LZO DLL, so I assumed you'd just copied LZO's code into paktool's source, which the GPL doesn't allow unless you licence the whole derivative work under GPL also (which is why some commentators have described it as "viral", and one microsoft exec described it as "cancer").

but it's easily avoidable by linking against a separate LZO DLL and providing that DLL separately.

Aruki

2015-01-19 05:19:17 pm

I compiled it as a static library. I guess I'll make it a DLL next time.

DJGrenola

Edit history:

DJGrenola: 2015-01-19 05:23:09 pm

2015-01-19 05:22:16 pm

licences are a bitch.

here you go.

(oh and technically I should have provided the full source code and a copy of the GPL with this post.)

attachments:

lzo2.dll

lzo2.lib

skysurf3000

Edit history:

skysurf3000: 2015-01-19 05:53:38 pm

2015-01-19 05:38:13 pm

I've just tried yEd, visualization is ok but heavily depends on how large you allow your graph to be... For anyone who wants to try it, the easiest way is to use the .tgf file format. Here is the example for the babygoth, which is quite self-explanatory:

1 Start
2 Patrol
3 Dead
4 PathFind
5 ProjectileAttack
6 TargetPatrol
7 Deactivate
8 Crouch
9 Generate
10 Attack
11 FollowPattern
12 Enraged
13 SpecialAttack
14 Approach
15 Taunt
16 SpecialAttack
17 ProjectileAttack
18 GetUp
19 TurnAround
20 TurnAround
#
1 2 HasPatrolPath
1 8 Default
2 15 InDetectionRange
2 15 SpotPlayer
4 12 AggressionCheck
4 19 LineOfSight
4 19 InRange
4 10 ShouldAttack
4 13 ShouldSpecialAttack
4 6 Leash
4 6 !InDetectionRange
4 14 OffLine
4 6 LostInterest
4 4 Delay
5 19 AnimOver
6 2 HasPatrolPath
6 2 InPosition
6 15 InDetectionRange
6 15 !LostInterest
6 7 InPosition
6 6 Delay
6 4 ShotAt
7 8 AnimOver
8 9 SpotPlayer
8 9 InDetectionRange
9 4 AnimOver
10 11 AnimOver
11 19 AnimOver
12 4 AnimOver
13 4 AnimOver
14 4 !OffLine
14 4 Delay
14 20 LineOfSight
14 20 Delay
14 16 ShouldSpecialAttack
14 17 ShouldFire
14 6 LostInterest
14 14 Delay
15 4 AnimOver
16 14 AnimOver
17 14 AnimOver
18 4 AnimOver
19 11 TooClose
19 10 ShouldAttack
19 13 ShouldSpecialAttack
19 5 ShouldFire
19 4 !InRange
19 4 Delay
19 4 !LineOfSight
19 19 Delay
20 16 ShouldSpecialAttack
20 17 ShouldFire
20 6 LostInterest
20 14 !LineOfSight
20 14 Delay
20 4 !OffLine
20 20 Delay

Quote from Paraxade:

Will look into it, but the point isn't just to create graphs, it's to be able to create an editor for the AI that can generate a graph based on the FSM data for the user to edit. Sounds like it'll be good for some ideas to see how it can be implemented, but not so useful to actually make any tools.

If what you want is a graphical editor to ASFM files then that is way above my skills.
You might still want to take a look at yEd though: there seems to be a lot of development-related documentation that may be useful.

I've just tried yEd, visualization is ok but heavily depends on how large you allow your graph to be... For anyone who wants to try it, the easiest way is to use the .tgf file format. Here is the example for the babygoth, which is quite self-explanatory:
[hidden]
1 Start
2 Patrol
3 Dead
4 PathFind
5 ProjectileAttack
6 TargetPatrol
7 Deactivate
8 Crouch
9 Generate
10 Attack
11 FollowPattern
12 Enraged
13 SpecialAttack
14 Approach
15 Taunt
16 SpecialAttack
17 ProjectileAttack
18 GetUp
19 TurnAround
20 TurnAround
#
1 2 HasPatrolPath
1 8 Default
2 15 InDetectionRange
2 15 SpotPlayer
4 12 AggressionCheck
4 19 LineOfSight
4 19 InRange
4 10 ShouldAttack
4 13 ShouldSpecialAttack
4 6 Leash 
4 6 !InDetectionRange
4 14 OffLine
4 6 LostInterest
4 4 Delay
5 19 AnimOver
6 2 HasPatrolPath
6 2 InPosition
6 15 InDetectionRange
6 15 !LostInterest
6 7 InPosition
6 6 Delay
6 4 ShotAt
7 8 AnimOver
8 9 SpotPlayer
8 9 InDetectionRange
9 4 AnimOver
10 11 AnimOver
11 19 AnimOver
12 4 AnimOver
13 4 AnimOver
14 4 !OffLine
14 4 Delay
14 20 LineOfSight
14 20 Delay
14 16 ShouldSpecialAttack
14 17 ShouldFire
14 6 LostInterest
14 14 Delay
15 4 AnimOver
16 14 AnimOver
17 14 AnimOver
18 4 AnimOver
19 11 TooClose
19 10 ShouldAttack
19 13 ShouldSpecialAttack
19 5 ShouldFire
19 4 !InRange
19 4 Delay
19 4 !LineOfSight
19 19 Delay
20 16 ShouldSpecialAttack
20 17 ShouldFire
20 6 LostInterest
20 14 !LineOfSight
20 14 Delay
20 4 !OffLine
20 20 Delay
[/hidden]

[quote="Paraxade"]Will look into it, but the point isn't just to create graphs, it's to be able to create an editor for the AI that can generate a graph based on the FSM data for the user to edit. Sounds like it'll be good for some ideas to see how it can be implemented, but not so useful to actually make any tools.
[/quote]

If what you want is a graphical editor to ASFM files then that is way above my skills. 
You might still want to take a look at yEd though: there seems to be a lot of development-related documentation that may be useful.

Aruki

2015-01-19 05:41:47 pm

Yeah... a graphical editor seems like the most intuitive interface for this kind of thing to me. It's not absolutely vital to have a visualization if it proves to be too complicated I suppose, I'm just thinking about what the ideal editor would be like. Thanks for pointing me to yEd though, I'll definitely check it out.

Bearborg

Edit history:

Bearborg: 2015-01-19 06:45:08 pm

2015-01-19 06:29:59 pm

Quote from Paraxade:

He's also okayed me to ask him a few more questions, not necessarily related to cut content. I don't want to bombard him with stuff so I'd like to just come up with a few good ones and then leave him alone for now. If anyone's got any suggestions what I could ask about, feel free to share, though remember he's an artist, so he's probably not going to know too many details about other areas of the game's development (like formats).

Well, there is something I've been curious about for a while. A "monkey lair", along with "monkey holes" and a "monkey beam", is mentioned in some of the Chozo Ruins and Phendrana Drifts concept sketches. The floor of the "monkey lair" seems identical to the floor of the room where the Incinerator Drone is fought, and Incinerator Drone's theme music has the filename "rui_monkeylower". So, were there going to be monkeys in the game, as some kind of recurring enemy?

Also, though this question is probably less interesting and kind of repetitive, the Phendrana Drifts art mentions "human amber glow prods" and "human fuel tanks". Was there supposed to be a human settlement on Tallon IV too?

[quote="Paraxade"]He's also okayed me to ask him a few more questions, not necessarily related to cut content. I don't want to bombard him with stuff so I'd like to just come up with a few good ones and then leave him alone for now. If anyone's got any suggestions what I could ask about, feel free to share, though remember he's an artist, so he's probably not going to know too many details about other areas of the game's development (like formats).[/quote]

Well, there is something I've been curious about for a while. A "monkey lair", along with "monkey holes" and a "monkey beam", is mentioned in some of the [url=http://vignette3.wikia.nocookie.net/metroid/images/7/79/Envir_sketches4.png/revision/latest]Chozo Ruins[/url] and [url=http://vignette4.wikia.nocookie.net/metroid/images/3/3c/Envir_sketches11.png/revision/latest]Phendrana Drifts[/url] concept sketches. The [url=http://vignette3.wikia.nocookie.net/metroid/images/f/f0/Envir_sketches12.png/revision/latest]floor of the "monkey lair"[/url] seems identical to the floor of the room where the Incinerator Drone is fought, and Incinerator Drone's theme music has the filename "rui_monkeylower". So, were there going to be monkeys in the game, as some kind of recurring enemy?

Also, though this question is probably less interesting and kind of repetitive, the Phendrana Drifts art mentions "human amber glow prods" and "human fuel tanks". Was there supposed to be a human settlement on Tallon IV too?

Antidote

2015-01-19 06:53:46 pm

I think it's in reference to the Etecoons, there are a few concepts of them in Prime i believe.

Aruki

2015-01-19 06:56:05 pm

Yeah, it is kinda weird but I'm not totally convinced it's referring to literal monkeys... I tend to not put much stock in the weird names developers pick for things sometimes, heh.

Antidote

2015-01-19 06:57:15 pm

The names for the objects should be a clue :P

Bearborg

2015-01-19 06:58:21 pm

Quote from Antidote:

I think it's in reference to the Etecoons, there are a few concepts of them in Prime i believe.

Are you sure you're not thinking of the "Endocoons" from Prime 3? As far as I'm aware, there's no artwork for them either, but they have scan text.

Antidote

2015-01-19 06:59:46 pm

You're probably right.

DJGrenola

Edit history:

DJGrenola: 2015-01-20 04:20:40 am

DJGrenola: 2015-01-20 04:19:20 am

2015-01-20 04:13:58 am

so here's a few things which might be useful in a future unpacking solution.

- automatic PAK file recognition through knowing a list of checksums of the PAK file headers (i.e. header + name table + entries table, checksumming the payloads themselves is overkill). retropak did this -- since zlib exports a crc32() function, I figured I might as well make use of it. here's the relevant function:

Code:

#define MP1NA000 "MP1 US 0-00"
#define MP1PAL   "MP1 PAL"
#define MP2NA    "MP2 US"
#define MP2PAL   "MP2 PAL"
static char *pakname_by_checksum (rpak_checksum_t cksum) {

  switch (cksum) {
  
    // mp1
    case 0x599f62d9: return "AudioGrp, "      MP1NA000;
    case 0x24faba9b: return "AudioGrp, "      MP1PAL;
    case 0x9715a862: return "GGuiSys, "       MP1NA000;
    case 0x0643b7a5: return "GGuiSys, "       MP1PAL;
    case 0x82b75cdf: return "Metroid1, "      MP1NA000;
    case 0xbf4892a5: return "Metroid1, "      MP1PAL;
    case 0x5cca900b: return "Metroid2, "      MP1NA000;
    case 0xd50bb89d: return "Metroid2, "      MP1PAL;
    case 0xed97b87e: return "Metroid3, "      MP1NA000;
    case 0x158577f1: return "Metroid3, "      MP1PAL;
    case 0xe4a67dd2: return "Metroid4, "      MP1NA000;
    case 0x08c5ec35: return "Metroid4, "      MP1PAL;
    case 0x6d9f5270: return "metroid5, "      MP1NA000;
    case 0x12c46850: return "metroid5, "      MP1PAL;
    case 0x479f4bbf: return "Metroid6, "      MP1NA000;
    case 0xd79fc3dd: return "Metroid6, "      MP1PAL;
    case 0xbaac3a0a: return "Metroid7, "      MP1NA000;
    case 0xf461a418: return "Metroid7, "      MP1PAL;
    case 0xa4996852: return "Metroid8, "      MP1NA000;
    case 0xf01c868d: return "Metroid8, "      MP1PAL;
    case 0xdefeb29f: return "MidiData, "      MP1NA000;
    case 0xcceaa877: return "MidiData, "      MP1PAL;
    case 0x007ec854: return "MiscData, "      MP1NA000;
    case 0x8f744f87: return "MiscData, "      MP1PAL;
    case 0xf1180d02: return "NoARAM, "        MP1NA000;
    case 0x47eab168: return "NoARAM, "        MP1PAL;
    case 0x26c7a8ff: return "SamGunFx, "      MP1NA000;
    case 0xa3fba426: return "SamGunFx, "      MP1PAL;
    case 0xf2030b89: return "SamusGun, "      MP1NA000 " or " MP1PAL;
    case 0x575f1615: return "TestAnim, "      MP1NA000 " or " MP1PAL;
    case 0xbcdc1a47: return "Tweaks, "        MP1NA000;
    case 0x9ee73643: return "Tweaks, "        MP1PAL;
    case 0xe11c9f6b: return "SlideShow, "     MP1NA000 " or " MP1PAL;

    // mp2
    case 0xee3d3e4b: return "AudioGrp, "      MP2NA  " or " MP2PAL;
    case 0xcf9a6318: return "FrontEnd, "      MP2NA;
    case 0x5df3f1f2: return "FrontEnd, "      MP2PAL;
    case 0xb006c47c: return "GGuiSys, "       MP2NA;
    case 0x8fd8f3b0: return "GGuiSys, "       MP2PAL;
    case 0xace5ad7a: return "LogBook, "       MP2NA;
    case 0x2e1e9511: return "LogBook, "       MP2PAL;
    case 0xd329f767: return "Metroid1, "      MP2NA  " or " MP2PAL;
    case 0x2a153f6b: return "Metroid2, "      MP2NA;
    case 0x4568dcaf: return "Metroid2, "      MP2PAL;
    case 0xc24bf5eb: return "Metroid3, "      MP2NA;
    case 0xd86a1099: return "Metroid3, "      MP2PAL;
    case 0x447d7432: return "Metroid4, "      MP2NA;
    case 0x0ce3c448: return "Metroid4, "      MP2PAL;
    case 0x9c126f84: return "Metroid5, "      MP2NA  " or " MP2PAL;
    case 0x5c64a0f3: return "Metroid6, "      MP2NA;
    case 0x605c462e: return "Metroid6, "      MP2PAL;
    case 0x230b691f: return "MidiData, "      MP2NA  " or " MP2PAL;
    case 0x1d07ea79: return "MiscData, "      MP2NA  " or " MP2PAL;
    case 0xd5a2ce33: return "NoARAM, "        MP2NA;
    case 0x7e52b571: return "NoARAM, "        MP2PAL;
    case 0x269291e0: return "SamGunFxLow, "   MP2NA  " or " MP2PAL;
    case 0x68ac4be2: return "SamGunFxMulti, " MP2NA  " or " MP2PAL;
    case 0xc5a83764: return "SamGunFx, "      MP2NA  " or " MP2PAL;
    case 0xb9dc4def: return "SamusGunLow, "   MP2NA  " or " MP2PAL;
    case 0x75aff537: return "SamusGun, "      MP2NA  " or " MP2PAL;
    case 0x8f9f1375: return "SlideShow, "     MP2NA  " or " MP2PAL;
    case 0x2c67be61: return "TestAnim, "      MP2NA  " or " MP2PAL;
    
    // others
    default: return "UNKNOWN";
  }
}

- automatic full-game unpacking. no reason why you should need a separate batch file or shell script to extract all the PAK files one by one, an unpacking tool should just know which files are in which game, and you should just be able to cd to the directory containing the PAKs and say like "myunpacker --mp1" and it will know the names of the files to unpack. it's not like those files are ever going to change. retropak didn't do this but I was annoyed with myself for missing it out. in fact you could even go so far as to have it autodetect the game based on the files it finds in the current directory, making the "--mp1" part unnecessary.

- long filenames that incorporate the names from the name tables where available. I mentioned this ages ago in this thread but it makes joyriding through a directory of extracted files quite a bit more meaningful for files that actually have name table entries.

- lower case extracted filenames. probably nobody but me cares about this but it saves lots of shift key mashing if you're working under a case sensitive filesystem (i.e. anything under Linux).

another thing retropak did was optionally to check that duplicate files within the PAK really are copies of one another by taking their checksums, again using the exported zlib crc32(). this was mainly for my own satisfaction when writing the thing because I was reversing the file format as I was writing retropak and I needed to prove to myself that the "duplicate" files really were duplicates. but still a nice feature for detecting possible file corruption.

so here's a few things which might be useful in a future unpacking solution.

- automatic PAK file recognition through knowing a list of checksums of the PAK file headers (i.e. header + name table + entries table, checksumming the payloads themselves is overkill). retropak did this -- since zlib exports a crc32() function, I figured I might as well make use of it. here's the relevant function:

[spoiler][code]
#define MP1NA000 "MP1 US 0-00"
#define MP1PAL   "MP1 PAL"
#define MP2NA    "MP2 US"
#define MP2PAL   "MP2 PAL"
static char *pakname_by_checksum (rpak_checksum_t cksum) {

switch (cksum) {
  
    // mp1
    case 0x599f62d9: return "AudioGrp, "      MP1NA000;
    case 0x24faba9b: return "AudioGrp, "      MP1PAL;
    case 0x9715a862: return "GGuiSys, "       MP1NA000;
    case 0x0643b7a5: return "GGuiSys, "       MP1PAL;
    case 0x82b75cdf: return "Metroid1, "      MP1NA000;
    case 0xbf4892a5: return "Metroid1, "      MP1PAL;
    case 0x5cca900b: return "Metroid2, "      MP1NA000;
    case 0xd50bb89d: return "Metroid2, "      MP1PAL;
    case 0xed97b87e: return "Metroid3, "      MP1NA000;
    case 0x158577f1: return "Metroid3, "      MP1PAL;
    case 0xe4a67dd2: return "Metroid4, "      MP1NA000;
    case 0x08c5ec35: return "Metroid4, "      MP1PAL;
    case 0x6d9f5270: return "metroid5, "      MP1NA000;
    case 0x12c46850: return "metroid5, "      MP1PAL;
    case 0x479f4bbf: return "Metroid6, "      MP1NA000;
    case 0xd79fc3dd: return "Metroid6, "      MP1PAL;
    case 0xbaac3a0a: return "Metroid7, "      MP1NA000;
    case 0xf461a418: return "Metroid7, "      MP1PAL;
    case 0xa4996852: return "Metroid8, "      MP1NA000;
    case 0xf01c868d: return "Metroid8, "      MP1PAL;
    case 0xdefeb29f: return "MidiData, "      MP1NA000;
    case 0xcceaa877: return "MidiData, "      MP1PAL;
    case 0x007ec854: return "MiscData, "      MP1NA000;
    case 0x8f744f87: return "MiscData, "      MP1PAL;
    case 0xf1180d02: return "NoARAM, "        MP1NA000;
    case 0x47eab168: return "NoARAM, "        MP1PAL;
    case 0x26c7a8ff: return "SamGunFx, "      MP1NA000;
    case 0xa3fba426: return "SamGunFx, "      MP1PAL;
    case 0xf2030b89: return "SamusGun, "      MP1NA000 " or " MP1PAL;
    case 0x575f1615: return "TestAnim, "      MP1NA000 " or " MP1PAL;
    case 0xbcdc1a47: return "Tweaks, "        MP1NA000;
    case 0x9ee73643: return "Tweaks, "        MP1PAL;
    case 0xe11c9f6b: return "SlideShow, "     MP1NA000 " or " MP1PAL;

// mp2
    case 0xee3d3e4b: return "AudioGrp, "      MP2NA  " or " MP2PAL;
    case 0xcf9a6318: return "FrontEnd, "      MP2NA;
    case 0x5df3f1f2: return "FrontEnd, "      MP2PAL;
    case 0xb006c47c: return "GGuiSys, "       MP2NA;
    case 0x8fd8f3b0: return "GGuiSys, "       MP2PAL;
    case 0xace5ad7a: return "LogBook, "       MP2NA;
    case 0x2e1e9511: return "LogBook, "       MP2PAL;
    case 0xd329f767: return "Metroid1, "      MP2NA  " or " MP2PAL;
    case 0x2a153f6b: return "Metroid2, "      MP2NA;
    case 0x4568dcaf: return "Metroid2, "      MP2PAL;
    case 0xc24bf5eb: return "Metroid3, "      MP2NA;
    case 0xd86a1099: return "Metroid3, "      MP2PAL;
    case 0x447d7432: return "Metroid4, "      MP2NA;
    case 0x0ce3c448: return "Metroid4, "      MP2PAL;
    case 0x9c126f84: return "Metroid5, "      MP2NA  " or " MP2PAL;
    case 0x5c64a0f3: return "Metroid6, "      MP2NA;
    case 0x605c462e: return "Metroid6, "      MP2PAL;
    case 0x230b691f: return "MidiData, "      MP2NA  " or " MP2PAL;
    case 0x1d07ea79: return "MiscData, "      MP2NA  " or " MP2PAL;
    case 0xd5a2ce33: return "NoARAM, "        MP2NA;
    case 0x7e52b571: return "NoARAM, "        MP2PAL;
    case 0x269291e0: return "SamGunFxLow, "   MP2NA  " or " MP2PAL;
    case 0x68ac4be2: return "SamGunFxMulti, " MP2NA  " or " MP2PAL;
    case 0xc5a83764: return "SamGunFx, "      MP2NA  " or " MP2PAL;
    case 0xb9dc4def: return "SamusGunLow, "   MP2NA  " or " MP2PAL;
    case 0x75aff537: return "SamusGun, "      MP2NA  " or " MP2PAL;
    case 0x8f9f1375: return "SlideShow, "     MP2NA  " or " MP2PAL;
    case 0x2c67be61: return "TestAnim, "      MP2NA  " or " MP2PAL;
    
    // others
    default: return "UNKNOWN";
  }
}[/code][/spoiler]
- automatic full-game unpacking. no reason why you should need a separate batch file or shell script to extract all the PAK files one by one, an unpacking tool should just know which files are in which game, and you should just be able to cd to the directory containing the PAKs and say like "myunpacker --mp1" and it will know the names of the files to unpack. it's not like those files are ever going to change. retropak didn't do this but I was annoyed with myself for missing it out. in fact you could even go so far as to have it autodetect the game based on the files it finds in the current directory, making the "--mp1" part unnecessary.

- long filenames that incorporate the names from the name tables where available. I mentioned this ages ago in this thread but it makes joyriding through a directory of extracted files quite a bit more meaningful for files that actually have name table entries.

- lower case extracted filenames. probably nobody but me cares about this but it saves lots of shift key mashing if you're working under a case sensitive filesystem (i.e. anything under Linux).

another thing retropak did was optionally to check that duplicate files within the PAK really are copies of one another by taking their checksums, again using the exported zlib crc32(). this was mainly for my own satisfaction when writing the thing because I was reversing the file format as I was writing retropak and I needed to prove to myself that the "duplicate" files really were duplicates. but still a nice feature for detecting possible file corruption.

Aruki

Edit history:

Aruki: 2015-01-20 05:49:14 am

Aruki: 2015-01-20 05:44:05 am

2015-01-20 05:39:42 am

Quote from DJGrenola:

- automatic PAK file recognition through knowing a list of checksums of the PAK file headers (i.e. header + name table + entries table, checksumming the payloads themselves is overkill). retropak did this -- since zlib exports a crc32() function, I figured I might as well make use of it. here's the relevant function:

erm... why is this useful?

Quote:

- automatic full-game unpacking. no reason why you should need a separate batch file or shell script to extract all the PAK files one by one, an unpacking tool should just know which files are in which game, and you should just be able to cd to the directory containing the PAKs and say like "myunpacker --mp1" and it will know the names of the files to unpack. it's not like those files are ever going to change. retropak didn't do this but I was annoyed with myself for missing it out. in fact you could even go so far as to have it autodetect the game based on the files it finds in the current directory, making the "--mp1" part unnecessary.

It's intended as a modding tool, so they actually can change (especially since you can change the world pak names to whatever you want without even touching the dol). I don't really think this is necessary. I like keeping it more general than that, and not having stuff super specific to each game's layout. Batch files get the same job done and they're pretty easy to use.

Actually I'll implement this as an optional feature, since yeah it would be nice to have so you can more easily specify unpacking options across the entire game; especially with DKCR and Tropical Freeze since they store paks in tons of different directories and it's honestly a pain to get to all of them.

Quote:

- long filenames that incorporate the names from the name tables where available. I mentioned this ages ago in this thread but it makes joyriding through a directory of extracted files quite a bit more meaningful for files that actually have name table entries.

I don't do this because the files themselves don't reference other files by their extracted names; the name tables are only used by the executable. If I'm reading a file and it references 158efe17.MLVL, but I actually have !1IntroLevel1027.MLVL, how am I supposed to find it? I could name the file "158efe17 !1IntroLevel1027.MLVL", but that still makes it more annoying than necessary to find the file.

I could maybe do this as an optional feature... I really don't want to make it the default, though.

Quote:

- lower case extracted filenames. probably nobody but me cares about this but it saves lots of shift key mashing if you're working under a case sensitive filesystem (i.e. anything under Linux).

PakTool already does this.

Quote:

another thing retropak did was optionally to check that duplicate files within the PAK really are copies of one another by taking their checksums, again using the exported zlib crc32(). this was mainly for my own satisfaction when writing the thing because I was reversing the file format as I was writing retropak and I needed to prove to myself that the "duplicate" files really were duplicates. but still a nice feature for detecting possible file corruption.

Hmm, not sure that situation should ever really come up but I guess that might be a nice thing to have?

[quote="DJGrenola"]- automatic PAK file recognition through knowing a list of checksums of the PAK file headers (i.e. header + name table + entries table, checksumming the payloads themselves is overkill). retropak did this -- since zlib exports a crc32() function, I figured I might as well make use of it. here's the relevant function:[/quote]

erm... why is this useful?

[quote]- automatic full-game unpacking. no reason why you should need a separate batch file or shell script to extract all the PAK files one by one, an unpacking tool should just know which files are in which game, and you should just be able to cd to the directory containing the PAKs and say like "myunpacker --mp1" and it will know the names of the files to unpack. it's not like those files are ever going to change. retropak didn't do this but I was annoyed with myself for missing it out. in fact you could even go so far as to have it autodetect the game based on the files it finds in the current directory, making the "--mp1" part unnecessary.[/quote]

It's intended as a modding tool, so they actually can change (especially since you can change the world pak names to whatever you want without even touching the dol). I don't really think this is necessary. I like keeping it more general than that, and not having stuff super specific to each game's layout. Batch files get the same job done and they're pretty easy to use.

Actually I'll implement this as an optional feature, since yeah it would be nice to have so you can more easily specify unpacking options across the entire game; especially with DKCR and Tropical Freeze since they store paks in tons of different directories and it's honestly a pain to get to all of them.

[quote]- long filenames that incorporate the names from the name tables where available. I mentioned this ages ago in this thread but it makes joyriding through a directory of extracted files quite a bit more meaningful for files that actually have name table entries.[/quote]

I don't do this because the files themselves don't reference other files by their extracted names; the name tables are only used by the executable. If I'm reading a file and it references 158efe17.MLVL, but I actually have !1IntroLevel1027.MLVL, how am I supposed to find it? I could name the file "158efe17 !1IntroLevel1027.MLVL", but that still makes it more annoying than necessary to find the file.

I could maybe do this as an optional feature... I really don't want to make it the default, though.

[quote]- lower case extracted filenames. probably nobody but me cares about this but it saves lots of shift key mashing if you're working under a case sensitive filesystem (i.e. anything under Linux).[/quote]

PakTool already does this.

[quote]another thing retropak did was optionally to check that duplicate files within the PAK really are copies of one another by taking their checksums, again using the exported zlib crc32(). this was mainly for my own satisfaction when writing the thing because I was reversing the file format as I was writing retropak and I needed to prove to myself that the "duplicate" files really were duplicates. but still a nice feature for detecting possible file corruption.[/quote]

Hmm, not sure that situation should ever really come up but I guess that might be a nice thing to have?

DJGrenola

2015-01-20 06:41:58 am

everything I suggested was intended as optional, not as default. they're just ideas that I felt it would be nice to have and didn't really require much in the way of extra developer effort.

Quote:

- automatic PAK file recognition through knowing a list of checksums of the PAK file headers (i.e. header + name table + entries table, checksumming the payloads themselves is overkill). retropak did this -- since zlib exports a crc32() function, I figured I might as well make use of it. here's the relevant function:

erm... why is this useful?

the thinking was that if you have two versions of prime 1 (NA + PAL) and two versions of prime 2 (NA + PAL) lying around on your machine and you end up in the wrong directory, it's nice to get confirmation that you're unpacking what you think you're unpacking. that's all. software such as MAME for example knows CRCs of all the ROMs it supports. not a big deal but the extra effort required to implement it seemed minimal to me since you already have the relevant data kicking around in memory and you get a free crc32() function thanks to zlib. but it's up to you.

Quote:

- automatic full-game unpacking. no reason why you should need a separate batch file or shell script to extract all the PAK files one by one, an unpacking tool should just know which files are in which game, and you should just be able to cd to the directory containing the PAKs and say like "myunpacker --mp1" and it will know the names of the files to unpack. it's not like those files are ever going to change. retropak didn't do this but I was annoyed with myself for missing it out. in fact you could even go so far as to have it autodetect the game based on the files it finds in the current directory, making the "--mp1" part unnecessary.

It's intended as a modding tool, so they actually can change (especially since you can change the world pak names to whatever you want without even touching the dol). I don't really think this is necessary. I like keeping it more general than that, and not having stuff super specific to each game's layout. Batch files get the same job done and they're pretty easy to use.

I guess might be nice as an optional feature, especially with DKCR and Tropical Freeze since they store paks in tons of different directories and it's honestly a pain to find all of them. Not going to make it do that by default though.

it's useful if you're starting out unpacking a whole game, that's all. and yes, obviously it would be optional. I'm aware you can create PAKs with different names; I wasn't suggesting that this should replace the ability to unpack a specific file by name, since that's clearly a subset of the functionality I proposed anyway. batch files aren't cross-platform, but then neither is paktool so I suppose it doesn't matter. again, it's up to you.

Quote:

- long filenames that incorporate the names from the name tables where available. I mentioned this ages ago in this thread but it makes joyriding through a directory of extracted files quite a bit more meaningful for files that actually have name table entries.

I don't do this because the files themselves don't reference other files by their extracted names; the name tables are only used by the executable. If I'm reading a file and it references 158efe17.MLVL, but I actually have !1IntroLevel1027.MLVL, how am I supposed to find it? I could name the file "158efe17 !1IntroLevel1027.MLVL", but that still makes it more annoying than necessary to find the file.

I could maybe do this as an optional feature... I really don't want to make it the default, though.

yes, retropak started the name with the file ID and then appended the name table entry. and yes, it was optional. we're seeing this from different viewpoints I think. you're seeing it from the viewpoint of someone who is expecting his own editor to be reading the files, and viewed from that perspective the referencing thing is a fair point. I'm seeing it from the viewpoint of someone who is interested in poking around on his own terms. I had a much more meaningful time looking through lists of extracted files that had their names right there in the filename. but again, it's your software. I don't really care because I have my own which already does this.

Quote:

- lower case extracted filenames. probably nobody but me cares about this but it saves lots of shift key mashing if you're working under a case sensitive filesystem (i.e. anything under Linux).

PakTool already does this.

I guess I was thinking more in terms of if you were going the long filename route, as having uppercase stuff in the filenames was annoying for me. IIRC paktool uses upper case extensions, but it's a very minor point anyway.

Quote:

another thing retropak did was optionally to check that duplicate files within the PAK really are copies of one another by taking their checksums, again using the exported zlib crc32(). this was mainly for my own satisfaction when writing the thing because I was reversing the file format as I was writing retropak and I needed to prove to myself that the "duplicate" files really were duplicates. but still a nice feature for detecting possible file corruption.

Hmm, not sure that situation should ever really come up but I guess that might be a nice thing to have?

IMO this is actually the least useful suggestion I made, lol. you'd never want this by default because it would hurt extraction speed substantially having to CRC all those duplicates. it was more just something that I did to convince myself that the duplicates were actually duplicates and with the code already there I just added it to retropak as an option.

everything I suggested was intended as optional, not as default. they're just ideas that I felt it would be nice to have and didn't really require much in the way of extra developer effort.

[quote][quote]- automatic PAK file recognition through knowing a list of checksums of the PAK file headers (i.e. header + name table + entries table, checksumming the payloads themselves is overkill). retropak did this -- since zlib exports a crc32() function, I figured I might as well make use of it. here's the relevant function:[/quote]

erm... why is this useful?[/quote]

the thinking was that if you have two versions of prime 1 (NA + PAL) and two versions of prime 2 (NA + PAL) lying around on your machine and you end up in the wrong directory, it's nice to get confirmation that you're unpacking what you think you're unpacking. that's all. software such as MAME for example knows CRCs of all the ROMs it supports. not a big deal but the extra effort required to implement it seemed minimal to me since you already have the relevant data kicking around in memory and you get a free crc32() function thanks to zlib. but it's up to you.

[quote][quote]- automatic full-game unpacking. no reason why you should need a separate batch file or shell script to extract all the PAK files one by one, an unpacking tool should just know which files are in which game, and you should just be able to cd to the directory containing the PAKs and say like "myunpacker --mp1" and it will know the names of the files to unpack. it's not like those files are ever going to change. retropak didn't do this but I was annoyed with myself for missing it out. in fact you could even go so far as to have it autodetect the game based on the files it finds in the current directory, making the "--mp1" part unnecessary.[/quote]

It's intended as a modding tool, so they actually can change (especially since you can change the world pak names to whatever you want without even touching the dol). I don't really think this is necessary. I like keeping it more general than that, and not having stuff super specific to each game's layout. Batch files get the same job done and they're pretty easy to use.

I guess might be nice as an optional feature, especially with DKCR and Tropical Freeze since they store paks in tons of different directories and it's honestly a pain to find all of them. Not going to make it do that by default though.[/quote]

it's useful if you're starting out unpacking a whole game, that's all. and yes, obviously it would be optional. I'm aware you can create PAKs with different names; I wasn't suggesting that this should replace the ability to unpack a specific file by name, since that's clearly a subset of the functionality I proposed anyway. batch files aren't cross-platform, but then neither is paktool so I suppose it doesn't matter. again, it's up to you.

[quote][quote]- long filenames that incorporate the names from the name tables where available. I mentioned this ages ago in this thread but it makes joyriding through a directory of extracted files quite a bit more meaningful for files that actually have name table entries.[/quote]

I don't do this because the files themselves don't reference other files by their extracted names; the name tables are only used by the executable. If I'm reading a file and it references 158efe17.MLVL, but I actually have !1IntroLevel1027.MLVL, how am I supposed to find it? I could name the file "158efe17 !1IntroLevel1027.MLVL", but that still makes it more annoying than necessary to find the file.

I could maybe do this as an optional feature... I really don't want to make it the default, though.[/quote]

yes, retropak started the name with the file ID and then appended the name table entry. and yes, it was optional. we're seeing this from different viewpoints I think. you're seeing it from the viewpoint of someone who is expecting his own editor to be reading the files, and viewed from that perspective the referencing thing is a fair point. I'm seeing it from the viewpoint of someone who is interested in poking around on his own terms. I had a much more meaningful time looking through lists of extracted files that had their names right there in the filename. but again, it's your software. I don't really care because I have my own which already does this.

[quote][quote]- lower case extracted filenames. probably nobody but me cares about this but it saves lots of shift key mashing if you're working under a case sensitive filesystem (i.e. anything under Linux).[/quote]

PakTool already does this.[/quote]

I guess I was thinking more in terms of if you were going the long filename route, as having uppercase stuff in the filenames was annoying for me. IIRC paktool uses upper case extensions, but it's a very minor point anyway.

[quote][quote]another thing retropak did was optionally to check that duplicate files within the PAK really are copies of one another by taking their checksums, again using the exported zlib crc32(). this was mainly for my own satisfaction when writing the thing because I was reversing the file format as I was writing retropak and I needed to prove to myself that the "duplicate" files really were duplicates. but still a nice feature for detecting possible file corruption.[/quote]

Hmm, not sure that situation should ever really come up but I guess that might be a nice thing to have?[/quote]

IMO this is actually the least useful suggestion I made, lol. you'd never want this by default because it would hurt extraction speed substantially having to CRC all those duplicates. it was more just something that I did to convince myself that the duplicates were actually duplicates and with the code already there I just added it to retropak as an option.

Aruki

Edit history:

Aruki: 2015-01-20 07:28:40 am

Aruki: 2015-01-20 07:28:11 am

2015-01-20 06:58:16 am

Quote from DJGrenola:

batch files aren't cross-platform, but then neither is paktool so I suppose it doesn't matter.

It doesn't rely on anything Windows specific (at least the rewrite won't) so there's not really any reason it couldn't be cross-platform. Batch files are nice though, especially in the scenario that Retro releases a new game that uses the same pak formats, or that there's a different version/new rerelease with changed names, or that a new prototype leaks with different pak names (the Echoes demo has a bunch of paks that don't exist on the final for instance).

That said, yeah, I'll implement this. Mostly because when working with Tropical Freeze recently, when I make a change to my unpacking setup, I have to go around and unpack every single pak again, and it's a giant pain in the ass.

Quote:

yes, retropak started the name with the file ID and then appended the name table entry. and yes, it was optional. we're seeing this from different viewpoints I think. you're seeing it from the viewpoint of someone who is expecting his own editor to be reading the files, and viewed from that perspective the referencing thing is a fair point. I'm seeing it from the viewpoint of someone who is interested in poking around on his own terms. I had a much more meaningful time looking through lists of extracted files that had their names right there in the filename. but again, it's your software. I don't really care because I have my own which already does this.

Yeah, I suppose that's a fair point... I'll consider it, not sure about that though. I usually just use the list dumps if I want to read the filename tables.

Quote:

I guess I was thinking more in terms of if you were going the long filename route, as having uppercase stuff in the filenames was annoying for me. IIRC paktool uses upper case extensions, but it's a very minor point anyway.

Ahh, figured you were just referring to A-F being lowercase in the file IDs. It does extract uppercase file extensions. I'm probably gonna keep them uppercase since I'm used to them by now, plus it's more accurate to the actual file. :P

Quote:

IMO this is actually the least useful suggestion I made, lol. you'd never want this by default because it would hurt extraction speed substantially having to CRC all those duplicates. it was more just something that I did to convince myself that the duplicates were actually duplicates and with the code already there I just added it to retropak as an option.

yeah haha I wasn't sure what kind of impact it would have on performance. Prolly skipping that one then.

The idea I have for a longer-term editing solution is at some point we'd be using customized project files rather than files directly extracted from the pak (because Retro definitely didn't do serious work with this file structure/name scheme and we shouldn't either). At that point it'd be more useful to be able to take a pak and convert all the files within it to the customized project formats, and then it'd be a lot easier to utilize stuff like the pak name tables (plus other names that you can grab from various files, like the node names from the ANCS files). But that's for when the editor is a lot further along, not now.

EDIT: to be honest, a GUI for PakTool wouldn't be the worst idea. It would let me fairly easily allow customization of all the things you're suggesting. I also like the idea of how easy it would make it to set up repacking once dependency parsing is working. I wouldn't remove command line functionality though, too useful.

[quote="DJGrenola"]batch files aren't cross-platform, but then neither is paktool so I suppose it doesn't matter.[/quote]

It doesn't rely on anything Windows specific (at least the rewrite won't) so there's not really any reason it couldn't be cross-platform. Batch files are nice though, especially in the scenario that Retro releases a new game that uses the same pak formats, or that there's a different version/new rerelease with changed names, or that a new prototype leaks with different pak names (the Echoes demo has a bunch of paks that don't exist on the final for instance).

That said, yeah, I'll implement this. Mostly because when working with Tropical Freeze recently, when I make a change to my unpacking setup, I have to go around and unpack every single pak again, and it's a giant pain in the ass.

[quote]yes, retropak started the name with the file ID and then appended the name table entry. and yes, it was optional. we're seeing this from different viewpoints I think. you're seeing it from the viewpoint of someone who is expecting his own editor to be reading the files, and viewed from that perspective the referencing thing is a fair point. I'm seeing it from the viewpoint of someone who is interested in poking around on his own terms. I had a much more meaningful time looking through lists of extracted files that had their names right there in the filename. but again, it's your software. I don't really care because I have my own which already does this.[/quote]

Yeah, I suppose that's a fair point... I'll consider it, not sure about that though. I usually just use the list dumps if I want to read the filename tables.

[quote]I guess I was thinking more in terms of if you were going the long filename route, as having uppercase stuff in the filenames was annoying for me. IIRC paktool uses upper case extensions, but it's a very minor point anyway.[/quote]

Ahh, figured you were just referring to A-F being lowercase in the file IDs. It does extract uppercase file extensions. I'm probably gonna keep them uppercase since I'm used to them by now, plus it's more accurate to the actual file. :P

[quote]IMO this is actually the least useful suggestion I made, lol. you'd never want this by default because it would hurt extraction speed substantially having to CRC all those duplicates. it was more just something that I did to convince myself that the duplicates were actually duplicates and with the code already there I just added it to retropak as an option.[/quote]

yeah haha I wasn't sure what kind of impact it would have on performance. Prolly skipping that one then.

The idea I have for a longer-term editing solution is at some point we'd be using customized project files rather than files directly extracted from the pak (because Retro definitely didn't do serious work with this file structure/name scheme and we shouldn't either). At that point it'd be more useful to be able to take a pak and convert all the files within it to the customized project formats, and then it'd be a lot easier to utilize stuff like the pak name tables (plus other names that you can grab from various files, like the node names from the ANCS files). But that's for when the editor is a lot further along, not now.

EDIT: to be honest, a GUI for PakTool wouldn't be the worst idea. It would let me fairly easily allow customization of all the things you're suggesting. I also like the idea of how easy it would make it to set up repacking once dependency parsing is working. I wouldn't remove command line functionality though, too useful.

Lord Fredrik

2015-01-20 10:19:09 am

So, Tropical Freeze DOES use PAK files. Coolio. I've been trying to find that out for days..

Antidote

2015-01-20 09:46:06 pm

Eh, it does, sort of. The pak format is similar, but there are some very important differences. I.E every pak starts with something like this:

Code:

00000000  52 46 52 4d 00 00 00 00  04 19 9f 6c 00 00 00 00  |RFRM.......l....|
00000010  00 00 00 00 50 41 43 4b  00 00 00 00 00 00 00 00  |....PACK........|
00000020  52 46 52 4d 00 00 00 00  00 01 ac aa 00 00 00 00  |RFRM............|
00000030  00 00 00 00 54 4f 43 43  00 00 00 01 00 00 00 01  |....TOCC........|
00000040  41 44 49 52 00 00 00 00  00 00 e6 5c 00 00 00 01  |ADIR.......\....|
00000050  00 00 00 00 00 00 00 00  00 00 06 66 47 45 4e 50  |...........fGENP|

Antidote

2015-01-21 12:36:14 am

RetroView (my viewer), just recently got a major overhaul, expect the source to be on github in a few days (after I get materials going again).

Lord Fredrik

2015-01-21 06:55:55 pm

I was not aware of an app named RetroView. Should be interesting.

Antidote

Edit history:

Antidote: 2015-01-24 11:51:04 am

Antidote: 2015-01-24 11:51:03 am

Antidote: 2015-01-22 08:21:10 am

Antidote: 2015-01-22 02:25:20 am

Antidote: 2015-01-22 02:21:05 am

Antidote: 2015-01-22 02:20:40 am

Antidote: 2015-01-22 02:18:56 am

2015-01-22 02:18:05 am

Just wrote a new pak extractor and it's exponentially faster than the original:

Quote from My Terminal:

[antidote@the-key-bearer build-pakextract-Desktop-Debug]$ time ~/Dropbox/mpakdump/mpakdump -f Metroid2.pak -q
real 0m1.789s
user 0m0.167s
sys 0m0.787s
[antidote@the-key-bearer build-pakextract-Desktop-Debug]$ time ./pakextract Metroid2.pak
real 0m0.052s
user 0m0.003s
sys 0m0.003s

Not really a great benchmark tool, but for something this simple it works fine. I'll have the source up in a few minutes.

edit:
Oh, further details: Both tools were on the same non-SSD drive as the pak, depending on your hardware setup your mileage may very. On my terabyte, mpakdump took a whopping 14 seconds on average, while my new pak extractor averaged 5.

edit 2:
Here is the source, again no windows binary:
--redacted--

edit 3:
Here is a windows version, again thanks parax for compiling it for me:
--redacted--

EDIT4:
--BUG FOUND--
Read this for details.

Just wrote a new pak extractor and it's exponentially faster than the original:
[quote="My Terminal"]
[antidote@the-key-bearer build-pakextract-Desktop-Debug]$ time ~/Dropbox/mpakdump/mpakdump -f Metroid2.pak -q
real	0m1.789s
user	0m0.167s
sys	0m0.787s
[antidote@the-key-bearer build-pakextract-Desktop-Debug]$ time ./pakextract Metroid2.pak 
real	0m0.052s
user	0m0.003s
sys	0m0.003s
[/quote]

Not really a great benchmark tool, but for something this simple it works fine. I'll have the source up in a few minutes.

edit:
Oh, further details: Both tools were on the same non-SSD drive as the pak, depending on your hardware setup your mileage may very. On my terabyte, mpakdump took a whopping 14 seconds on average, while my new pak extractor averaged 5.

edit 2:
Here is the source, again no windows binary:
--redacted--

edit 3:
Here is a windows version, again thanks parax for compiling it for me:
--redacted--

EDIT4:
--BUG FOUND--
Read [url=https://m2k2.taigaforum.com/post/metroid_prime_hacking_research_864.html#metroid_prime_hacking_research_865-]this[/url] for details.

Aruki

2015-01-22 08:56:08 am

I think we should probably start looking into documenting formats a little better... I know there's a few people watching this thread who are active on Metroid wikis; can any of you point us to a place where we could document info on file formats, etc. in a wiki format? I've looked into Wikia a little and it seems like it'd work ok I guess, but if there's a better option I'd rather use that. (Preferably we could put up info on DKCR/DKCTF too since they're by the same developer and use similar formats...)

I always liked the way the Custom Mario Kart Wiiki documents formats (example) so it'd be cool if we could document Retro formats in a similar fashion. I think this thread's grown large enough that nobody new is going to want to dig through the whole thing to find info though. :P

Antidote

Edit history:

Antidote: 2015-01-22 09:34:22 am

Antidote: 2015-01-22 09:34:08 am

2015-01-22 09:30:44 am

Oh, apparently I figured this format out a LONG time ago, and completely forgot about it:

Code:

#include "common-types.bt"
 
BigEndian();
 
// mark used bytes with a light green background
SetBackColor(cLtGreen);
 
struct Unk
{
    int   txtrID <format = hex>;
    float unkFloat;
    int   unk4;
    int   unk5;
    int   unk6;
    int   unk7;
    float unkFloat2;
};
 
enum ChunkIdent
{
    ANCS = 0x414E4353,
    ANIM = 0x414E494D,
    CMDL = 0x434D444C,
    CINF = 0x43494E46,
    CSKR = 0x43534B52,
    SNFO = 0x534E464F,
    STRG = 0x53545247
};
 
struct ScanChunk
{
    ChunkIdent ident <format = hex>;
    if (ident == SNFO)
    {
        ushort dataLen;
        uchar data[dataLen];
    }
    else
    {
        if (fileInfo.snfoChunk.dataLen < 0x0251)
            uint assetId <format = hex>;
        else
            uint64 assetId <format = hex>;
    };
};
 
struct scan_file
{
    uint32 __magic_test <hidden = true>;
    FSeek(-4);
    if (__magic_test != 0x5343414E)
    {
        // Metroid Prime 1
        uint32 fileVersion<format = hex>;
        uint32 magic <format = hex>;
        if (magic != 0x0BADBEEF)
            return;
 
        uint32 frmeID <format = hex>;
        uint32 stringID <format = hex>;
        uint32  unk1;
        uint32  unk2;
        ubyte   padding;
        if (padding != 0)
            Printf("Padding != 0\n");
        Unk     unk[4];
    }
    else
    {
        // Metroid Prime 2
        uint magic <format = hex>;
        uint version;
        ubyte unk1;
        uint  unk2;
        ScanChunk snfoChunk <optimize = false>;
        uint chunkCount;
        ScanChunk chunks2[chunkCount] <optimize = false>;
    }
};
 
 
struct scan_file fileInfo;

EDIT:
darn paraxade spoiling my fun :P, this is the SCAN format used to describe logbook entries.

Oh, apparently I figured this format out a LONG time ago, and completely forgot about it:
[code]
#include "common-types.bt"
 
BigEndian();
 
// mark used bytes with a light green background
SetBackColor(cLtGreen);
 
struct Unk
{
    int   txtrID <format = hex>;
    float unkFloat;
    int   unk4;
    int   unk5;
    int   unk6;
    int   unk7;
    float unkFloat2;
};
 
enum ChunkIdent
{
    ANCS = 0x414E4353,
    ANIM = 0x414E494D,
    CMDL = 0x434D444C,
    CINF = 0x43494E46,
    CSKR = 0x43534B52,
    SNFO = 0x534E464F,
    STRG = 0x53545247
};
 
struct ScanChunk
{
    ChunkIdent ident <format = hex>;
    if (ident == SNFO)
    {
        ushort dataLen;
        uchar data[dataLen];
    }
    else
    {
        if (fileInfo.snfoChunk.dataLen < 0x0251)
            uint assetId <format = hex>;
        else
            uint64 assetId <format = hex>;
    };
};
 
struct scan_file
{
    uint32 __magic_test <hidden = true>;
    FSeek(-4);
    if (__magic_test != 0x5343414E)
    {
        // Metroid Prime 1
        uint32 fileVersion<format = hex>;
        uint32 magic <format = hex>;
        if (magic != 0x0BADBEEF)
            return;
 
        uint32 frmeID <format = hex>;
        uint32 stringID <format = hex>;
        uint32  unk1;
        uint32  unk2;
        ubyte   padding;
        if (padding != 0)
            Printf("Padding != 0\n");
        Unk     unk[4];
    }
    else
    {
        // Metroid Prime 2
        uint magic <format = hex>;
        uint version;
        ubyte unk1;
        uint  unk2;
        ScanChunk snfoChunk <optimize = false>;
        uint chunkCount;
        ScanChunk chunks2[chunkCount] <optimize = false>;
    }
};
 
 
struct scan_file fileInfo;
[/code]

EDIT:
darn paraxade spoiling my fun :P, this is the SCAN format used to describe logbook entries.

embyr_75

2015-01-22 12:35:40 pm

Metroid Wiki

Quote from Paraxade:

I think we should probably start looking into documenting formats a little better... I know there's a few people watching this thread who are active on Metroid wikis; can any of you point us to a place where we could document info on file formats, etc. in a wiki format? I've looked into Wikia a little and it seems like it'd work ok I guess, but if there's a better option I'd rather use that. (Preferably we could put up info on DKCR/DKCTF too since they're by the same developer and use similar formats...)

I always liked the way the Custom Mario Kart Wiiki documents formats (example) so it'd be cool if we could document Retro formats in a similar fashion. I think this thread's grown large enough that nobody new is going to want to dig through the whole thing to find info though. :P

If you're looking into independent wikis, off the top of my head, the existing tcrf wiki might work. It's not limited to particular games and deals with hacking elements. MetroidWiki.org could host the info for you, if you wanted.

If you want to start up a whole new wiki just dedicated to Retro formats, you could go with a wiki farm (like Wikia), but I'd suggest going to NIWA Network forums and asking for guidance. It's the wiki alliance metroid wiki is part of and they have a supportive user base with experience getting new wikis off the ground. They'd probably love that idea. There are also people there willing to host for you, if you look into that option.

[quote="Paraxade"]I think we should probably start looking into documenting formats a little better... I know there's a few people watching this thread who are active on Metroid wikis; can any of you point us to a place where we could document info on file formats, etc. in a wiki format? I've looked into Wikia a little and it seems like it'd work ok I guess, but if there's a better option I'd rather use that. (Preferably we could put up info on DKCR/DKCTF too since they're by the same developer and use similar formats...)

I always liked the way the Custom Mario Kart Wiiki documents formats ([url=http://wiki.tockdom.com/wiki/MDL0_(File_Format)]example[/url]) so it'd be cool if we could document Retro formats in a similar fashion. I think this thread's grown large enough that nobody new is going to want to dig through the whole thing to find info though. :P[/quote]

If you're looking into independent wikis, off the top of my head, the existing tcrf wiki might work. It's not limited to particular games and deals with hacking elements. MetroidWiki.org could host the info for you, if you wanted.

If you want to start up a whole new wiki just dedicated to Retro formats, you could go with a wiki farm (like Wikia), but I'd suggest going to [url=http://www.niwanetwork.org]NIWA Network[/url] forums and asking for guidance. It's the wiki alliance metroid wiki is part of and they have a supportive user base with experience getting new wikis off the ground. They'd probably love that idea. There are also people there willing to host for you, if you look into that option.

Aruki

Edit history:

Aruki: 2015-01-22 01:38:01 pm

Aruki: 2015-01-22 01:33:53 pm

Aruki: 2015-01-22 01:12:33 pm

Aruki: 2015-01-22 01:01:52 pm

Aruki: 2015-01-22 12:56:25 pm

2015-01-22 12:55:26 pm

... I just remembered that m2k2 has a couple wikis already. Maybe we should look into putting stuff there, lol. That seems like the best/easiest solution.

Looks like the existing ones are just specifically for Prime 2/3, so maybe we'd need a new one created? Will need to ask the staff about it.

EDIT: Also, I'd be totally fine with using MetroidWiki personally, it just seems like adding DK stuff would end up going out of the scope of that wiki a bit... do you think so, embyr? We could try putting DK stuff on the DK wiki but splitting the info like that seems like an unnecessary pain when there's a lot of similar formats between them.