<- 1  -   of 75 ->
^^
vv
List results:
Search options:
Use \ before commas in usernames
Edit history:
SkippyJr: 2015-01-30 10:48:33 pm
EVNT file format.  Almost entirely unknown, but parsable.  One part of it looks like it references file id's, which could aid in cracking the meanings.

long format

if format == 1
long Num_type0's
  that many struct type0
long Num_type4's
  that many struct type4
long Num_type5's
  that many struct type5

if format == 2
long Num_type0's
  that many struct type0
long Num_type6's
  that many struct type6
long Num_type5's
  that many struct type5
long Num_type8's
  that many struct type8

----

common bits:
  short 0001
  null termated ascii string
  short type ( 0000, 0004, 0005, 0006, 0007, 0008)
  float ?
  long ? (0 or 2)
  long counter (eg if there are 4 of them they are numbered 0,1,2,3)
  byte ? 01
  float ?
  long ? (-1 or 0)
  long ? (0 or # in type 5's)

struct type0:
  <common bits>
  byte ? 01

struct type4:
  <common bits>
  long ?
  null terminated ascii string

struct type5:
  <common bits>
  long ? size? index?
  long 'PART' or 'ELSC'
  long fileid
  null terminated ascii string
  float ?
  long ? (0 or 2)

struct type6: (== type7 as seen in one file)
  <common bits>
  long ? (a type5 # ref?)
  null terminated ascii string

struct type8:
  <common bits>
  long ? (0 or 2)
  float ?
  float ?

Edit: added skysurf3000's observations. A 32-bit long also has the same common caller in all, moved to <common bits>.
Oh sweet. Do you think you could do the same thing for ANIM as well? I think I got confused trying to figure out ANIM's structure.

Actually, there's still another section at the end of ANCS we can't quite figure out as well (that one's a bitch, eh?). It's the table at the end of the file, after the last node.
Edit history:
skysurf3000: 2015-01-30 05:45:22 am
Nice work on the EVNT file! here is what i could saw:
In the common bits: I think i saw a 07 short type in a struct type4 in 2114e16b.EVNT (in Metroid2).
In struct type 5, right before the ascii string, there is always a "PART" or "ELSC", followed by the id of a .PART or .ELSC file.
I also think a few strings may be bone names (would that make sense?), but I have never been able to find which .CINF they would come from. Maybe decoding the end of the ANCS could help, since ANCS link both to CINF and EVNT.

By the way, what are you guys using to do this type of thing? I have a basic hex viewer but that's about it...
Sooo, Parax, Sinistar and I were all going back and forth about AGSC, Sinistar was insisting that it matches some files in the MusyX player gcm found online. Parax and I were doubtful, suddenly he sends us files, and boom all doubts were erased: Turns out AGSC is an amalgam of MusyX's  different files, pool, proj, samp, and sdir, with CSNG being a 1:1 MusyX sequence file. All that effort to crack it and it was already partially done for us. How irritating >.>
Edit history:
Aruki: 2015-01-31 08:21:10 pm
yeah, and I found debug strings that get printed when the game reads the different AGSC data chunks. They're called (in order) Pool, Project, Sample, and Sample Directory, which matches the musyx files. Soo pretty annoying we're only now finding out that it's actually a common format, after having already cracked half of it ourselves. Oh well.

EDIT: though to clarify, there doesn't seem to be much in the way of documentation for these formats currently, so our work wasn't necessarily in vain anyway. Hopefully this extra bit of insight is what's needed to crack the rest of the file.
Hopefully figured out the rest of ANCS making it parsable, going through the disassembly helped.  As earlier parts of ANCS did unexpected things when various parameters were set, the same may yet happen here.

Again meanings unknown.  One interesting thing is whenever a float is read it and the following long are read by the same calling functions, so they are supposed to be together.
after last node:

short ?? (0004 usually, 0003)

long count
repeat X count:
  string
  long ? 0
  long ? fileid?
  long counter from 0 to count-1
  string
  float ? 0
  long ? 2  (float+long paired)

long count
repeat X count:
  long ?
  long ?
  long ?
  long type
  struct that type

(one more time without the 3 longs)
long type
struct that type

(may be optional or may always be here???)
long ? 0
float ?
float ?
long ? 0

optional at the end (dunno how to know if it will be there):
long count
repeat X count:
  long ? fileid1?
  long ? fileid2?

---------------------------

struct type0:
  long ? 0
  long ? fileid?
  long ?
  string
  float ?
  long ?  (float+long paired)

struct type1:
  float ?
  long ? 0 (paired)
  byte 00
  byte 01
  long ?  ( seems to = 3 just before the long/float/float/long section)

struct type3:
  <NULL>

other types unknown

@skysurf3000: That type7 in EVNT you found might be a mistake or maybe the "type" is actually a list of flags which rarely change.  It is parsed exactly the same as a type6, as in same functions are involved.  As for tools, you can use doltool to convert the DOL file to an ELF file and then use (a crosscompiled) objdump or your favorite disassembly program to look at the assembly.  Someone without much programming experience could learn how magic(5) files are created to parse files with the file(1) command under linux/etc.
Man that first pattern seems obvious now in hindsight. Can't believe I didn't pick up on that.

I'm trying to follow along, but the pattern seems to break in Metroid1/27550808.ANCS, at 0x2366. Any idea why?
Edit history:
Antidote: 2015-02-01 12:13:29 am
I seem to have figured out that pattern:
Code:
struct Anim
{
    uint animID <format = hex>;
    uint unk7;
    string  Name2;
    float unk4;
    uint unk5;
    uint unk6;
};

struct UnkData
{
    string Name;
    uint type <format = hex>;
    if (type != 0)
    {
        uint subAnimationCount;
        uint unk;
    }
    if (type != 0)
        Anim anim[subAnimationCount - 1]  <optimize = false>;
    uint animID <format = hex>;
    uint index;
    string Name2;
    float unk4;
    uint unk5;
};

This has worked on every file i've tried (about 90% of Metroid1)
Edit history:
Antidote: 2015-02-01 01:59:27 am
Antidote: 2015-02-01 01:40:25 am
skippy, that first short is the amount of sections to follow, we're nearly there only a couple of ANCS files break now.

8169653A.ANCS <- that one breaks on the second section.

EDIT:
Just finished testing Metroid1.pak, that file is the only one that breaks in it.
Quote from SkippyJr:
optional at the end (dunno how to know if it will be there):
long count
repeat X count:
  long ? fileid1?
  long ? fileid2?


The first seems to always be an ANIM file, and the second an EVNT file.
Moreover the ID of the EVNT file is repeated in position 8 of the ANIM file.
In UnkData if type != 0 (4 in the Paraxade's file) what is listed as "uint unk and uint unk6" is another type (the same type just as before).  So a type 0 (struct Anim) is embedded in a type 4.  When type == 0 what follows is a struct Anim.  (These are 1st section types, not 2nd section types.)  There is a case statement in the code (at 802e0000) looking for types 0 through 5.  Its possible that there could be recursion like a type 0 (struct Anim) in a type4 in a type4.  Here's how it looks like it works:

1st section

long count
repeat X count:
  string
  long typeA
  struct that type

struct type0 = struct Anim
  long animID
  long counter from 0 to count-1 only on outermost?
  string
  float ? 0
  long ? 2 (float+long paired)

struct type4
  long countA
  repeat X countA:
    long typeA (0 only seen so far)
    struct typeA

Other structs unknown.

Quote from Antidote:
8169653A.ANCS <- that one breaks on the second section.

This one has two things more:
- a struct of type 2 (which is functionally identical to type1 except the last long is read in a different place and thus has a different meaning)
- the long/float/float/long section has a non-zero value

long ?
float ?
float ?
long count
repeat X count
  long ?
  long type  (always type 3 in that file, the NULL type)
  struct that type

Knowing that that is a struct is because the same call from 802e1e10 (and its related case statement) is made in that section too.
Alright. I'm gonna try to get this working with my dependency parser today. Hopefully if there's any more unknown structures in the format, they'll come up during a full parse.
Edit history:
Aruki: 2015-02-01 05:18:41 pm
Aruki: 2015-02-01 04:39:02 pm
Aruki: 2015-02-01 04:26:32 pm
Unrelatedly, Ninji is awesome so animation modes 4/5 are cracked! This mode is a lot more complicated than mode 2... while 2 is just "multiply the time by a scale factor and add an offset", 4/5 are a little more similar to an actual animation. I'm calling the four parameters: scale, numFrames, step, and offset. scale roughly controls the animation playback speed. numFrames is how many frames the animation loops through before it resets back to the beginning. step is the amount the animation scrolls by each frame. offset is.. pretty much useless, since it's applied to the time value at the beginning of the calculation, so it doesn't really have much of an impact on how the animation turns out in the end. I'm not sure what the purpose of that one is.

The math to calculate the offset is basically:

Code:
float value = step * scale * (offset + s);
float uv_offset = (float)(short)(float)(numFrames * fmod(value, 1.0f)) * step;


4/5 both calculate the offset the same way. The difference between the two is where in the texture matrix the resulting offset is placed; 4 will apply it to U, and 5 will apply it to V.

Echoes uses mode 2 for almost all UV scrolls, so it's not as important in that game. Prime 1 uses 4/5 for most everything though, so the animations don't really work without supporting this one.
Edit history:
Antidote: 2015-02-01 06:03:23 pm
Nice, I should get back and recreate my PART template, can't believe I lost that thing by formatting /home D:
The format is fairly simple, yet at the same time annoyingly complex, you have a series of FOURCC "tokens" which define certain behavior, such as creating an emitter, changing the color, using models/textures as particles, environment dynamics (wind, gravity, etc.) The overall particle system is pretty extensive, and it'd take ages to go through the whole thing.
Edit history:
Antidote: 2015-02-02 02:45:50 am
Well, sinistar and I think we might have figured out most of the MusyX development stack, for one thing I think Retro used one massive project that had each different type of SFX separated via groups, and subgroups, then converted and "cooked" into the AGSC using their development tools. The only real mystery left is what command 104 is, so far it's the only unknown command left. There is also some things we still don't fully understand about the .proj file.
Custom Sounds are a thing!!

Edit history:
Aruki: 2015-02-02 06:41:59 pm
Aruki: 2015-02-02 03:08:29 pm
Aruki: 2015-02-02 03:01:02 pm
Aruki: 2015-02-02 10:06:57 am
Aruki: 2015-02-02 10:06:33 am
More UV animation stuff. I think UV animations are just supposed to transform UV coordinates in general; they don't necessarily need to be actual animations, per se. It turns out that the reflective spheremaps are actually done through a UV animation - mode 0. I've sorta been cheating on that to get it working correctly, by hardcoding a case in the shader for when the vertex normal is used as a texgen input, but now that I'm realizing what the correct way to do it is I've sorta gotten sidetracked trying to figure out how to implement it correctly. :P

Essentially, modes 2-5 generate a texture matrix, and modes 0/1/6/7 generate both a texture matrix and a post-transform matrix. texgen can then be set to use the matrices; so it'd take the input attribute, then optionally, multiply by a texture matrix, normalize, and/or multiply by a post transform matrix. Stuff that doesn't use any matrix transformations just uses an identity matrix for the texture/post-transform matrices.

Mode 0's texture matrix is inverse(view matrix) * model matrix (no translation).

By the way, modes 0/1/7 all operate on the model/view matrices. 6 seems to operate on the model matrix, but not the view matrix. Mode 7 takes two float parameters; the rest take none.

EDIT: by the way, the Exterior Docking Hangar force fields that MrSinistar just demonstrated in his video up there... the flash effect they do when you shoot them is actually baked in as a UV animation! So the force field is actually constantly flashing, but ingame it normally isn't visible unless you've shot it, so it seems like a reaction to being shot. Needless to say it makes me glad I added a button to turn off the UV animation playback.
Edit history:
skysurf3000: 2015-02-02 06:11:07 pm
skysurf3000: 2015-02-02 05:55:44 pm
I don't know if anyone took the time to completely parse MLVL files, so here you go:
// HEADER //
magic : DEAFBABE;
int always_17; (a version number?)
strg_id;
savw_id;
cmdl_id; //some of them can be omitted and are replaced by 0xFFFFFFFF

int unknown_struct_count;
typedef struct {
short room_id;
short param1;
short room_id;
short param2;
short often4;
byte often0;
}UNKNOWN;
UNKNOWN [unknwon_struct_count];

// BODY //

int room_count;
int const_1;

typedef struct {
strg_id;
float[18];
mrea_id;
float? ;
int short_count;
short [short_count];
int always_0;

int file_count;
typedef struct{
file_id ;
file_type; (4CC)
}FILE;
FILE [count_files];

int int_counter;
int [int_counter];

int unknown_struct_2_count;
typedef struct {
int pair_int_nbr;
typedef struct {
int int1;
int int2;
} PAIR_INT;
PAIR_INT [pair_int_nbr];

int always_4;
float [12];
}UNKNOWNED_STRUCT_2;
UNKNOWNED_STRUCT_2 [float_list_count];

}ROOM;

ROOM [room_count];

// FOOTER //

mapw_id;
byte always_0;
int always0;

int pair_count;
typedef struct {
int;
float?;
}PAIR;
PAIR [pair_count];

byte always0;

int count_triples;
typedef struct {
int triple_int;
int oftenFFFFFFFF;
int oftenFFFFFFFF;
} TRIPLE;
TRIPLE triple[count_triples];

int layer_count;
string [layer_count]; // 0 terminated

int room_count_again;
int default_layer[room_count_again];


About the unknown structures:
- in the first one, I'm not sure if it is 5 short or 2 int + 1 short. But I noticed that what i marked as room_id seem to always be equal and never go over the number of rooms, so 5shorts would make sense. Moreover when you go through the list the room_ids always increase.
- in the PAIR structure, the int is always increasing.

Also a huge number of layers are called "don't load" ^^
There are also a number of funy ones:
"REMOVE THE SPIKES!!!!"
"HIDE THE SPIKES!!"
"remove this boulder when destroyed"
"1st pass Pirates" followed by  "2nd Pass Ninjas"
"Irving" and "Amanda"
"JUMP BLOCKER"
Edit history:
Aruki: 2015-02-02 06:34:31 pm
Yep, Miles has had MLVL mostly cracked for a while... he's been working on getting a Wiki page up for it but he's not done as of yet. There's not too many unknowns in there left, I think. The first unknown struct is Memory Relays, the float[18] is a transform matrix/bounding box for the aether box, the short count is a list of connected room IDs IIRC, and I think unknown_struct_2 is connected rooms. The triples are actually one long and one 64-bit set of bit flags that sets which layers are enabled by default on a new save. There's one of those per room and the long is the number of layers in that room.

That wiki page really needs to go up, I'll bug him about it soon.

Also, Irving and Amanda would be referring to a couple of the developers: Irving Mah (Engineer) and Amanda Rubright (Designer). Probably they're the ones who worked on those layers.
Arf well I had fun anyway ^^

Do you have any idea of how the game links a layer to its actual content?
Edit history:
Aruki: 2015-02-04 01:24:50 am
You mean the names? I'm not sure. Miles knows, though.

EDIT: The MLVL page is up on the wiki.
Edit history:
skysurf3000: 2015-02-04 05:35:09 am
skysurf3000: 2015-02-04 05:31:32 am
Great thanks!
My question is how does the game knows which enemies,... (objects?) should be (or not) inside a given layer?
What I find confusing is that the dependency list is made for every area and not for every layer.
Maybe this kind of data is somewhere in the MREA file?

By the way, is the wiki up-to-date concerning the MREA?

EDIT:
I'm also wondering if it could be a good idea to extract some data from the files into something "human-readable".
A few examles:
- knowing which AFSM file corresponds to which enemy,
- having the list of layers of a room and which objects are within which layer
- knwowing which SCAN file is linked to what
A general idea is to have a "reverse dependency table", that is a table that tells you what are the files that depend on one file.

So two questions:
- Would someone be interested in that?
- Where to put it? A wiki? A big google spredsheet?
Quote from Paraxade:
Also, Irving and Amanda would be referring to a couple of the developers: Irving Mah (Engineer) and Amanda Rubright (Designer). Probably they're the ones who worked on those layers.

Both of those layers are empty.

Quote from skysurf3000:
Great thanks!
My question is how does the game knows which enemies,... (objects?) should be (or not) inside a given layer?
What I find confusing is that the dependency list is made for every area and not for every layer.

The dependency list is divided by layer; you use the offsets after the list to find the start of the files for each layer. The actual object lists are part of the MREA, in the SCLY.
Edit history:
Aruki: 2015-02-04 10:43:32 pm
Aruki: 2015-02-04 10:31:13 pm
Quote from MilesSMB:
Quote from Paraxade:
Also, Irving and Amanda would be referring to a couple of the developers: Irving Mah (Engineer) and Amanda Rubright (Designer). Probably they're the ones who worked on those layers.

Both of those layers are empty.


Well then I guess they didn't work very hard on them :P Maybe it was temporary stuff?

Quote from skysurf3000:
By the way, is the wiki up-to-date concerning the MREA?


Part of it is. There's no articles for collision or SCLY yet (both of which are largely known), but the rest is mostly there for MP1 at least.

Quote:
I'm also wondering if it could be a good idea to extract some data from the files into something "human-readable".
A few examles:
- knowing which AFSM file corresponds to which enemy,
- having the list of layers of a room and which objects are within which layer
- knwowing which SCAN file is linked to what
A general idea is to have a "reverse dependency table", that is a table that tells you what are the files that depend on one file.

So two questions:
- Would someone be interested in that?
- Where to put it? A wiki? A big google spredsheet?


If you mean a listing of what the files in the paks actually are so we have a reference when looking for specific assets, yeah, that would be a nice thing to have on the wiki. About a year ago Miles and I put together a spreadsheet on Google Docs to do this for models and MREAs: https://docs.google.com/spreadsheets/d/1yY19ZuNfCuHw3cgNVEfKvlUTR3G6HGMb9hM9DNdxxhw/edit?usp=drive_web

It would be preferable to have it on the wiki, though, and to extend it to other types of files as well.

EDIT: on an unrelated note, PWE now supports playback of every material animation mode, except 7. I'm not sure I'm going to bother with 7 for a while because from what I saw of it it includes basically all the most confusing parts from all the other modes, and I've seen barely anything that uses it anyway. In any case, everything I know about all of them is up on the wiki now.
RetroView also supports all but 7, if someone is interested in figuring it out, we'd greatly appreciate it, otherwise it's just not used enough to go through the trouble.