<- 1  -   of 75 ->
^^
vv
List results:
Search options:
Use \ before commas in usernames
DJGrenola
2014-09-04 12:09:03 pm
But that's equally true of 32 bit IDs. You can just CRC32 the assets. Also I'm not convinced that's how it was done because every time an asset was edited its ID would change.
Ntsc
2014-09-04 02:04:22 pm
More entropy?  I don't really know. Usually hashes are on file names not content. Allows multiple tools to use file names and then hash for faster lookup times and storage space.
Antidote
2014-09-04 02:15:52 pm
The IDs are based on a string hash from what I can tell, the reason they went with a larger asset id is relatively unknown.
DJGrenola
2014-09-04 02:15:52 pm
I don't know, I mean I'm guessing it was done for some sort of nebulous 'future proofing' but it seems weird that they would deliberately sacrifice RAM and performance for no actual apparent gain, specially when you consider that they modified the font format between mp1 and 2 to save RAM.

Another question is why the PAL loader is slower for prime 1. I haven't ripped my PAL copy yet so maybe there's more data but I'd like to know the answer.
Antidote
Antidote
Edit history:
Antidote: 2014-09-04 02:19:28 pm
2014-09-04 02:17:46 pm
It's due to LZO being a poorer compression algorithm, it tends to be a tad slow to decompress compared to huffman trees.
It's probably also due to the fact that the maps are segmented and that requires extra logic to perform.
DJGrenola
2014-09-04 02:19:19 pm
Is LZO used in PAL prime 1? I thought it was still zlib, as I say I haven't bothered to check yet
Antidote
Antidote
Edit history:
Antidote: 2014-09-04 02:23:21 pm
Antidote: 2014-09-04 02:22:16 pm
Antidote: 2014-09-04 02:20:49 pm
2014-09-04 02:20:12 pm
oh misread that, I read that as MP2, the reason the PAL loader in prime 1 is slower is because it's more reliable.
The NTSC and 00-0 loader is faster, but less reliable. The Players choice version also uses the PAL loader, which fixes the crashes in elevators.
DJGrenola
DJGrenola
Edit history:
DJGrenola: 2014-09-04 02:39:08 pm
2014-09-04 02:37:23 pm
Ah that's true. It did fix the crashes. I wonder whether it was a zlib bug or something that they weren't able to fix because zlib was an off-the-shelf thing they didn't write themselves. I should check the zlib versions in the two DOLs to see if they updated it. Also maybe they did something like using a smaller decompression buffer size which would explain why it was a bit slower. It's something we wondered about endlessly back in the day.
Aruki
2014-09-04 06:31:36 pm
It's possible they used the names... after all, they never bothered changing the Ingsmasher's name to something other than "Elite Space Pirate" and so its model has the same asset ID as the MP1 Elite Pirate

Could you test that theory using the Corruption proto? There's a bunch of text files that contain the filepaths and the original names of all the assets. It'd be interesting if it was possible to generate the same ID they did using that.
Antidote
2014-09-04 10:09:25 pm
I'll take a look.
Antidote
2014-09-04 10:26:25 pm
I'm not coming up with anything viable :/
Aruki
Aruki
Edit history:
Aruki: 2014-09-05 07:16:14 am
Aruki: 2014-09-05 07:15:17 am
Aruki: 2014-09-05 06:21:01 am
2014-09-05 06:16:01 am
As a side note, working on reimplementing object support (this is basically a rewrite). I'm finding it neat how properly rendering the materials is making pickups stand out a lot more than they did before. Here's a "before" shot from the old version for comparison.



edit: Here's another picture now that I've fixed some bugs:



Because I like comparisons
Drewseph
2014-09-05 11:08:07 am
You know this got me thinking.  How many bosses can be active in one room?
BioSpark
2014-09-05 12:17:05 pm
what are the samuses? spawn points?
MilesSMB
2014-09-05 12:21:59 pm
Cutscene actors (same as Ridley).
MrSinistar
MrSinistar
Edit history:
MrSinistar: 2014-09-05 09:50:11 pm
2014-09-05 09:47:16 pm
Incredible work, Paraxade.  Just when I think it can't get any better, I get hit with more awesome screenshots and progress. lol

I'm messing around with the sound effects from Metroid Prime 3 and CSMP seems to be a bit easier to understand than AGSC (for me at least).  The CSMPs in the Prototype E3 Demo are different in the header chunk but other than that, they're identical in both data and filenames.

Here's what a CSMP header looks like from the Prototype.  This file is 02AF2A3D9CA9E062.csmp aka ui_x_download_lp_00:

Code:
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000  43 53 4D 50 00 00 00 01 4E 41 4D 45 00 00 00 18  CSMP....NAME....
00000010  75 69 2F 75 69 5F 78 5F 64 6F 77 6E 6C 6F 61 64  ui/ui_x_download
00000020  5F 6C 70 5F 30 30 00 00 49 4E 46 4F 00 00 00 0C  _lp_00..INFO....
00000030  01 01 00 00 00 00 00 00 42 C8 00 00 44 41 54 41  ........BÈ..DATA
00000040  00 00 73 24 00 00 C8 C0 00 00 00 00 00 00 7D 00  ..s$..ÈÀ......}.
00000050  00 01 00 00 00 00 00 00 00 00 C8 BF 00 00 00 00  ..........È¿....
00000060  01 3C 04 9F 0B E2 FB 3F 07 D3 FE F8 0E AB F8 B7  .<.Ÿ.âû?.Óþø.«ø·
00000070  04 50 02 D3 0C AE FA E0 09 87 FE 0D 0E BF F8 E3  .P.Ó.®úà.‡þ..¿øã
00000080  00 00 00 47 00 00 00 00 00 47 00 00 00 00 00 00  ...G.....G......
00000090  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000000A0  00 00 00 00                                      ....


And here's the same CSMP in the final game:

Code:
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000  43 53 4D 50 00 00 00 01 49 4E 46 4F 00 00 00 0C  CSMP....INFO....
00000010  01 01 00 00 00 00 00 00 42 C8 00 00 50 41 44 20  ........BÈ..PAD 
00000020  00 00 00 14 FF FF FF FF FF FF FF FF FF FF FF FF  ....ÿÿÿÿÿÿÿÿÿÿÿÿ
00000030  FF FF FF FF FF FF FF FF 44 41 54 41 00 00 73 24  ÿÿÿÿÿÿÿÿDATA..s$
00000040  00 00 C8 C0 00 00 E5 6E 00 00 7D 00 00 01 00 00  ..ÈÀ..ån..}.....
00000050  00 00 00 00 00 00 C8 BF 00 00 00 00 01 3C 04 9F  ......È¿.....<.Ÿ
00000060  0B E2 FB 3F 07 D3 FE F8 0E AB F8 B7 04 50 02 D3  .âû?.Óþø.«ø·.P.Ó
00000070  0C AE FA E0 09 87 FE 0D 0E BF F8 E3 00 00 00 47  .®úà.‡þ..¿øã...G
00000080  00 00 00 00 00 47 00 00 00 00 00 00 00 00 00 00  .....G..........
00000090  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................


In every CSMP I've seen so far, the first 4 bytes after "DATA" always point to the 5th byte of the 6th to last row of the file.  In the examples above, these four bytes will point to 0x7324.  I'm assuming that this is either a loop point or perhaps signaling the end of audio data that the game should play.  For example, in impacts/sam_r_painsm_00a2 AKA EB7A03A6DAFD9F1D.CSMP, the DATA header offset just points to a bunch of zeros at the end of the file.  Meanwhile, in the prototype and final versions of "ui_x_download_lp_00" they point to the same offset, but they point to different parts of the audio stream...4 bytes ahead to be exact.  The raw stream data is identical in both files, but it's been shifted 4 bytes thanks due to the fact that the header is different in the final game and shifted all the data after it.

The INFO header looks to be exactly the same for every single file.  And after the INFO header, there's a PAD header for padding, I assuming, to replace that missing NAME header.  The padding also helps make all the other CSMP files line up really nicely in the hex editor which makes comparing files much easier.

In the finalized version of CSMP, the audio stream looks like it always starts somewhere in row 0xA0 and the byte in 0x7F always seems to match the first byte of the audio stream.

Row 0x40 looks like to be possibly offsets or other important ADPCM data.  A couple of the bytes don't seem to change over several files, mainly 0x4A which is always "7D".  Hope this info helps! :)
Aruki
2014-09-06 01:00:21 am
Interesting work there, MrSinistar. The only thing I can see to add to that is, if you haven't already realized, the value that says 0x42C80000 in the INFO section is a float, and its value is 100.0.
Antidote
Antidote
Edit history:
Antidote: 2014-09-06 02:10:55 am
2014-09-06 02:10:39 am
I've been taking a look at CAUD, and it has a reference to CSMP, so my hunch on CAUD being midis seems fairly appropriate. Nice work on CSMP, I'll add those notes to my CSMP Template and see what I can make of it.
MrSinistar
2014-09-06 02:27:23 am
Quote from Paraxade:
Interesting work there, MrSinistar. The only thing I can see to add to that is, if you haven't already realized, the value that says 0x42C80000 in the INFO section is a float, and its value is 100.0.


Thanks!  I don't know how to look up floats, so I'm glad you spotted that!

Quote from Antidote:
I've been taking a look at CAUD, and it has a reference to CSMP, so my hunch on CAUD being midis seems fairly appropriate. Nice work on CSMP, I'll add those notes to my CSMP Template and see what I can make of it.


Thanks, Antidote!  I think you're absolutely right.  CAUD is most likely volume settings, pitch bends, etc.
Aruki
2014-09-06 02:28:32 am
Google for a hex-to-float converter; they're pretty useful.
Antidote
Antidote
Edit history:
Antidote: 2014-09-06 02:34:46 am
Antidote: 2014-09-06 02:32:14 am
Antidote: 2014-09-06 02:30:53 am
2014-09-06 02:29:59 am
Any hex editor worth it's salt will have one included though, but for quick references it'll be a good idea to have one bookmarked. Basically when looking at a float value for a 3D game, if it has an exponent it's probably not a valid float.

For example:
0.25 <- this would be a valid value, while
-2.532179e+38 <- this would not

Also, if you see a bunch of hex values starting with 0x3F you're probably looking at a float, 0x3F800000 is 1.0f.
MrSinistar
MrSinistar
Edit history:
MrSinistar: 2014-09-06 03:40:11 am
MrSinistar: 2014-09-06 03:34:29 am
MrSinistar: 2014-09-06 03:33:08 am
MrSinistar: 2014-09-06 03:30:05 am
MrSinistar: 2014-09-06 03:17:30 am
MrSinistar: 2014-09-06 03:16:52 am
2014-09-06 03:16:14 am
Good to know, thanks a lot guys :)

Some further developements or rather revisions to previous discoveries here: the 0x4A byte does change but I only encountered it once; in the scan text beep sound effect aka 4F85A11A10FC9A2E.CSMP.

Also, that DATA EOF offset header points to the 5th byte of the FOURTH to last row of audio stream data.  Some sound files put in one or two rows of FF bytes and those seem to be just padding.  Examples below:

Samus Death Scream / 4A66553F9D4BC355.csmp

Code:
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000  43 53 4D 50 00 00 00 01 49 4E 46 4F 00 00 00 0C  CSMP....INFO....
00000010  01 00 00 00 00 00 00 00 42 C8 00 00 50 41 44 20  ........BÈ..PAD 
00000020  00 00 00 14 FF FF FF FF FF FF FF FF FF FF FF FF  ....ÿÿÿÿÿÿÿÿÿÿÿÿ
00000030  FF FF FF FF FF FF FF FF 44 41 54 41 00 00 C8 04  ÿÿÿÿÿÿÿÿDATA..È.

. . .

0000C800  20 00 E1 00 00 3F F3 E0 60 1E 00 E2 0C 30 C4 0D   .á..?óà`..â.0Ä.
0000C810  40 11 F0 10 00 10 F1 0E 50 20 E0 1F 00 F2 FE 40  @.ð...ñ.P à..òþ@
0000C820  60 B3 3C 22 D1 0E 20 E1 00 00 FF 0F F0 0F 10 02  `³<"Ñ. á..ÿ.ð...
0000C830  20 FF 10 00 00 00 00 00 00 00 00 00 00 00 00 00   ÿ..............

EOF


And here's the Scan Visor Download sound loop, 02AF2A3D9CA9E062.csmp:

Code:
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000  43 53 4D 50 00 00 00 01 49 4E 46 4F 00 00 00 0C  CSMP....INFO....
00000010  01 01 00 00 00 00 00 00 42 C8 00 00 50 41 44 20  ........BÈ..PAD 
00000020  00 00 00 14 FF FF FF FF FF FF FF FF FF FF FF FF  ....ÿÿÿÿÿÿÿÿÿÿÿÿ
00000030  FF FF FF FF FF FF FF FF 44 41 54 41 00 00 73 24  ÿÿÿÿÿÿÿÿDATA..s$

. . . 

00007320  15 B6 E5 52 36 60 54 F3 34 20 D2 6B 3C 0F 5A 1B  .¶åR6`Tó4 Òk<.Z.
00007330  04 AF 3D E8 9C CE C1 D7 14 E6 B6 70 10 D3 F4 87  .¯=èœÎÁ×.æ¶p.Óô‡
00007340  55 F1 51 3E 6D 40 BD C0 75 80 1A 20 3E 14 13 F5  UñQ>m@½Àu€. >..õ
00007350  26 57 64 30 0F CD 9D 70 00 00 00 00 00 00 00 00  &Wd0.Í.p........
00007360  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
00007370  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ

EOF


Note the two rows of FF padding at the end of file.
Antidote
2014-09-06 03:57:16 am
Had a bit of a breakthrough with CAUD.
Code:
//--------------------------------------
//--- 010 Editor v5.0.1 Binary Template
//
// File:
// Author:
// Revision:
// Purpose:
//--------------------------------------


BigEndian();
struct CSMPInfo
{
    uint unk8;
    uint64 CSMPId <format = hex>;
    char unk[unk8 - 8];
};

struct
{
    uint magic <format = hex>;
    uint version;
    string name;
    ubyte unk1;
    uint unk2;
    ubyte unk3[3];
    uint unk4;
    float unkFloat1;
    float unkFloat2;
    uint unk6;
    uint unk7;
    CSMPInfo csmpInfo[unk7] <optimize = false>;
}file;
MrSinistar
2014-09-06 04:01:00 am
Rows 0x60 and 0x70 are very interesting.  I collected this data from five different CSMP files.

Code:
0D 5A F9 C0 0B 00 FC 84 0E 37 F9 AB 08 22 FF 74
0E 45 F9 66 0A F6 FC F0 0E 93 F9 66 00 00 00 00


Code:
0D 11 F9 23 08 3D FC AD 0F 05 F8 96 01 93 01 1F 
0E 22 F9 1D 0A 46 FD 0A 0E EC F8 F1 00 00 00 21


Code:
0B E2 FB 3F 07 D3 FE F8 0E AB F8 B7 04 50 02 D3 
0C AE FA E0 09 87 FE 0D 0E BF F8 E3 00 00 00 47


Code:
0B 77 FA B0 07 98 FE 17 0E BA F8 7B 05 FE FD A4 
0C 18 FB 32 08 9F FE 6E 0F 29 F8 64 00 00 00 00


Code:
0F 97 F8 3B 0D 5A FA 59 0F D9 F8 15 08 96 FF 0C
0F 57 F8 9C 0E 34 F9 AF 0F EE F8 0F 00 00 00 54


Every other byte seems to be of a similar pattern.  As you can see, the first byte is between 00-0F, then the third byte starts with F, then the five byte repeats the pattern again, back to the 0. This goes on until the last four bytes of each row.  Again the very last byte, located in offset 0x7F, is going to be the same byte at the beginning of the audio stream.  I don't know what the last four bytes in row 0x60 are.

Awesome work on CAUD, Antidote!
Antidote
Antidote
Edit history:
Antidote: 2014-09-06 04:05:11 am
2014-09-06 04:03:34 am
It definitely looks like there is information on pitch bending, but it appears to be on a PER SAMPLE basis rather than per loop :O
In regards to that pattern, it might be a coef LUT.

Wait, that last 4 bytes, that's a size value of some kind.