Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
project:sc2inspector [2011/12/29 06:52] – [Project Log] smark | project:sc2inspector [2012/01/02 12:27] – [Project Log] smark | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== SC2Inspector ====== | + | ====== SC2Inspector |
- | ===== Project Log ===== | + | ===== Reference |
- | === 12/28/2011 @ 23:46 === | + | |
- | Ok, so I've gotten a decent amount done. I was looking at code for an MQP parser and it looks like the SC2Replay files have a slightly different format. They start with MPQ\x1B then 1024 bytes into the file have another MPQ\x1A which actually starts the normal MPQ file. MPQ1B seems to be a StarCraft II only option for displaying additional metadata without having to read the file. | + | |
- | Using the following data taken from a random SC2Replay: | + | ==== MPQ\x1B Format ==== |
+ | Serialized Data (most up to date): | ||
+ | < | ||
+ | |--- 0 (BinaryData) - Starcraft II Replay 11 | ||
+ | |--- 1 (ArrayWithKeys) - Data Array | ||
+ | | |--- 0 (NumberInVLF) - Unknown | ||
+ | | |--- 1 (NumberInVLF) - Version: Major | ||
+ | | |--- 2 (NumberInVLF) - Version: Minor | ||
+ | | |--- 3 (NumberInVLF) - Version: Patch | ||
+ | | |--- 4 (NumberInVLF) - Version: Build | ||
+ | | `--- 5 (NumberInVLF) - Unknown, might be Revision? Seems close to Build | ||
+ | |--- 2 (NumberInVLF) - Unknown | ||
+ | `--- 3 (NumberInVLF) - Game Length (in 1/16 seconds)</ | ||
- | {{: | + | Extended information, |
- | + | ||
- | I was able to determine | + | |
^ Attribute | ^ Attribute | ||
- | | Id | x0-x3 | 4D50511B | + | | Id | x0-x3 | 4D 50 51 1B | MPQ\x1B |
- | | UserDataMaxSize | + | | UserDataMaxSize |
- | | HeaderOffset | + | | HeaderOffset |
- | | UserDataSize | + | | UserDataSize |
| DataType | | DataType | ||
| NumberOfElements | x17 | 08 | Indicates 4 elements in array (VLF) | | | NumberOfElements | x17 | 08 | Indicates 4 elements in array (VLF) | | ||
Line 20: | Line 28: | ||
| DataType | | DataType | ||
| NumberOfElements | x20 | 2C | Indicates 22 elements in the upcomming array | | | NumberOfElements | x20 | 2C | Indicates 22 elements in the upcomming array | | ||
- | | StarCraftII | + | | StarCraftII |
| Index | x43 | 02 | Sets index to 1 | | | Index | x43 | 02 | Sets index to 1 | | ||
| DataType | | DataType | ||
Line 39: | Line 47: | ||
| DataType | | DataType | ||
| Version | | Version | ||
- | | I'm unsure after this part. Nothing seems to add up correctly. |||| | + | | Unknown |
- | | Unknown | + | |
+ | ==== replay.details Format ==== | ||
+ | < | ||
+ | |--- 0 (SimpleArray) - Array of player structs | ||
+ | | `--- x (ArrayWithKeys) - Player struct | ||
+ | | |--- 0 (BinaryData) - Player name | ||
+ | | |--- 1 (ArrayWithKeys) - Probably some further details | ||
+ | | | ||
+ | | | ||
+ | | | ||
+ | | | ||
+ | | |--- 2 (BinaryData) - Localized race name | ||
+ | | |--- 3 (ArrayWithKeys) - Array of player color values | ||
+ | | | ||
+ | | | ||
+ | | | ||
+ | | | ||
+ | | |--- 4 (NumberInVLF) - Unknown | ||
+ | | |--- 5 (NumberInVLF) - Unknown | ||
+ | | |--- 6 (NumberInVLF) - Handicap | ||
+ | | |--- 7 (NumberInVLF) - Team | ||
+ | | `--- 8 (NumberInVLF) - Unknown | ||
+ | |--- 1 (BinaryData) - Localized map name | ||
+ | |--- 2 (BinaryData) - Unknown | ||
+ | |--- 3 (ArrayWithKeys) - Array containing map preview file names | ||
+ | | `--- 0 (BinaryData) - Map preview file name | ||
+ | |--- 4 (NumberOfOneByte) - Unknown | ||
+ | |--- 5 (NumberInVLF) - Save time of the replay | ||
+ | |--- 6 (NumberInVLF) - Unknown | ||
+ | |--- 7 (BinaryData) - Unknown | ||
+ | |--- 8 (BinaryData) - Unknown | ||
+ | |--- 9 (BinaryData) - Unknown | ||
+ | |--- 10 (SimpleArray) - Likely something about the map file | ||
+ | | |--- 0 (BinaryData) - Unknown | ||
+ | | `--- 1 (BinaryData) - Unknown | ||
+ | |--- 11 (NumberOfOneByte) - Unknown | ||
+ | |--- 12 (NumberInVLF) - Unknown | ||
+ | `--- 13 (NumberInVLF) - Unknown</ | ||
+ | |||
+ | ==== replay.attributes.events Sample Data ==== | ||
+ | |||
+ | | < | ||
+ | 0x1 - Part | ||
+ | 0x2 - Part | ||
+ | 0x01F4 | ||
+ | 0x1 - Humn | ||
+ | 0x2 - Humn | ||
+ | 0x0BB9 | ||
+ | 0x1 - Zerg | ||
+ | 0x2 - Terr | ||
+ | 0x07DC | ||
+ | 0x1 - T3 | ||
+ | 0x2 - T1 | ||
+ | 0x07E2 | ||
+ | 0x1 - T3 | ||
+ | 0x2 - T4 </ | ||
+ | 0x1 - T1 | ||
+ | 0x2 - T2 | ||
+ | 0x0BBB | ||
+ | 0x1 - 100 | ||
+ | 0x2 - 100 | ||
+ | 0x07D6 | ||
+ | 0x1 - T3 | ||
+ | 0x2 - T4 | ||
+ | 0x07D2 | ||
+ | 0x1 - T2 | ||
+ | 0x2 - T1 | ||
+ | 0x0BBA | ||
+ | 0x1 - tc04 | ||
+ | 0x2 - tc07</ | ||
+ | 0x1 - T1 | ||
+ | 0x2 - T2 | ||
+ | 0x07D4 | ||
+ | 0x1 - T1 | ||
+ | 0x2 - T2 | ||
+ | 0x07D3 | ||
+ | 0x1 - T2 | ||
+ | 0x2 - T2 | ||
+ | 0x0BBC | ||
+ | 0x1 - Medi | ||
+ | 0x2 - Medi | ||
+ | 0x07DB | ||
+ | 0x1 - T1 | ||
+ | 0x2 - T2</ | ||
+ | 0x1 - T1 | ||
+ | 0x2 - T2 | ||
+ | 0x03E8 | ||
+ | 0x10 - Dflt | ||
+ | 0x0BC0 | ||
+ | 0x1 - Obs | ||
+ | 0x2 - Obs | ||
+ | 0x0BC2 | ||
+ | 0x10 - yes | ||
+ | 0x07E1 | ||
+ | 0x1 - T3 | ||
+ | 0x2 - T4 | ||
+ | 0x07D0 | ||
+ | 0x10 - t2</ | ||
+ | 0x10 - Fasr | ||
+ | 0x0BBE | ||
+ | 0x10 - 10 | ||
+ | 0x0BC1 | ||
+ | 0x10 - Priv | ||
+ | 0x03E9 | ||
+ | 0x10 - yes | ||
+ | 0x07D1 | ||
+ | 0x10 - 1v1</ | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | | | ||
+ | |||
+ | ===== Project Log ===== | ||
+ | === 1/2/2011 @ 05:26 === | ||
+ | Ok so I've been working on this most of the day. I've finished the InitData file as well as the AttributesEvents file! It looks like I'm 100% done with the game metadata about the players. Here's all the information I have: {{: | ||
+ | |||
+ | The attributes.events file has kind of an interesting format. This one bit of code pretty much does all the work: | ||
+ | <code csharp> | ||
+ | uint AttribHeader; | ||
+ | uint AttribId; | ||
+ | int PlayerId; | ||
+ | string AttribVal; | ||
+ | int NumSlots; | ||
+ | for (int i = 0; i < NumAttribs; i++) { | ||
+ | AttribHeader = BinaryReader.ReadUInt32(); | ||
+ | AttribId = BinaryReader.ReadUInt32(); | ||
+ | PlayerId = BinaryReader.ReadByte(); | ||
+ | AttribVal = Conversion.ReverseString(Encoding.Default.GetString(BinaryReader.ReadBytes(4))).Replace(" | ||
+ | if (!AttribDict.ContainsKey(AttribId)) { | ||
+ | AttribDict.Add(AttribId, | ||
+ | } | ||
+ | AttribDict[AttribId].Add(PlayerId, | ||
+ | } | ||
+ | if (NumAttribs == 0) { | ||
+ | throw new Exception(" | ||
+ | }</ | ||
+ | |||
+ | I run that code after I read four bytes from the beginning. It splits everything out into what can be used as a multidimensional associative array. Here's some sample data: [[sc2inspector# | ||
+ | |||
+ | Committed r7. | ||
+ | |||
+ | === 12/31/2011 @ 03:12 === | ||
+ | Ok so I've been slacking on my documentation. I have fully parsed all of the documented fields in the replay.details file. Here is what the output looks like: {{: | ||
+ | |||
+ | Wow ok, so after a 45min battle of trying to get that file to upload with the new Dokuwiki install, we're back on track. I ran into an issue with ParseVLFNumber() where it wouldn' | ||
+ | |||
+ | I also modified InspectorViewModel and ReplayViewModel to allow for multiple replays to be loaded. I'm very happy with how fast the program is (not that the files are that big, but it is complex) at the moment. I've finished replay.details. I think I need to move on to InitData next, but I'm not sure. | ||
+ | |||
+ | Because I went back and modified MPQ\x1B UserData retrieval I was able to successfully extract the actual game playtime. The value that is given is in 1/16s of a second. I do not know where this value comes from. I do know that the current version of Sc2gears (8.8) is displaying incorrect game lengths. It seems to be applying ~1/22 of a second to the values. I verified this by viewing a replay in the StarCraft II client itself. The game time is 20:02 and SC2gears reports it as 14:28. Weird. | ||
+ | |||
+ | Committed r6. | ||
+ | |||
+ | === 12/31/2011 @ 00:16 === | ||
+ | I've taken some time to update the wiki page. I've added the references section and went back and converted the old MPQ\x1B information from raw byte " | ||
+ | |||
+ | === 12/30/2011 @ 20:22 === | ||
+ | I've been successful so far in decrypting the replay.details file. I spent some time and developed a serialized data parser. I've been able to piece together the format of the replay.details file: [[sc2inspector# | ||
+ | |||
+ | === 12/30/2011 @ 18:03 === | ||
+ | I've finally gotten to the point where I've read and enumerated all of the files in the MPQ. I think I did a really good job laying out my classes. I have a MPQArchive class which looks like this (displaying listfile contents): {{: | ||
+ | |||
+ | Now I have to determine what exactly is in each replay.* file. | ||
+ | |||
+ | ^ Filename ^ Purpose ^ | ||
+ | | replay.details | Basic metadata | | ||
+ | | replay.initData | Unknown | | ||
+ | | replay.attribute.events | Unknown | | ||
+ | | replay.game.events | Actions | | ||
+ | | replay.message.events | Chat, Ping | | ||
+ | | replay.smartcam.events | Presumably player cameras | | ||
+ | | replay.sync.events | Presumably consistency checks | | ||
+ | |||
+ | |||
+ | |||
+ | === 12/30/2011 @ 00:39 === | ||
+ | Ok well I took a short break to hang out with my roommates but I've finally be able to retrieve the raw uncompressed data for each file! I'm fairly sure this means that the next step is to start parsing the actual SC2 data! Right here I'd like to give a big shout out to Foole, the author of " | ||
+ | |||
+ | I've spent some time getting my comments up to date. I've also begun using SharpZipLib (http:// | ||
+ | |||
+ | Once again I'm looking over my code and it doesn' | ||
+ | |||
+ | I'm going to take a short break then see what I can decipher from the other replay.game.* files. Committed r5. | ||
+ | |||
+ | === 12/29/2011 @ 20:44 === | ||
+ | Well I've been working on this for about an hour and I decided to redo the block stuff. Now both the BlockTable and the HashTable are Hashtables with Hash and Block objects stored within. This makes it easier to resolve the HashTable entries to their BlockTable entries. I've been checking all my numbers against " | ||
+ | |||
+ | === 12/29/2011 @ 05:57 === | ||
+ | Finally making some promising progress! I believe I last left off where I had just finished parsing MPQ\x1B and the User Data. I've since made quite a bit of progress parsing MPQ\x1A (forever after known as "The MPQ File" | ||
+ | |||
+ | I've used more code than I'd like from other people, but most of this is bit-shifting magic that is beyond me. These being most of the Hash Table encryption/ | ||
+ | |||
+ | I feel like I should be writing more, but when I look over the code it's pretty simple in hindsight. It's simply using BinaryReader to read a bunch of numbers from a binary file. The only complex parts are the decryption parts which I find very confusing. I think I'll write a BROAD bit of pseudo-code about how this thing works: | ||
+ | |||
+ | < | ||
+ | Open SC2Replay rile | ||
+ | Read first 4 bytes | ||
+ | If first 4 bytes are MPQ\x1B Then | ||
+ | Read User Data from MPQ1B header (Version, etc), most importantly the Header Offset (usually 1024) | ||
+ | End If | ||
+ | Advance Byte Buffer to Header Offset (1024) | ||
+ | Read next 4 bytes | ||
+ | If first 4 bytes are MPQ\x1A Then | ||
+ | Read information about MPQ1A header (ArchiveSize, | ||
+ | End If | ||
+ | Advance Byte Buffer to beginning of HashTable | ||
+ | Read next (HashTableSize * 16) Bytes | ||
+ | Decrypt HashTable | ||
+ | Read the following over and over (16 bytes total) from the decrypted stream: Name1, Name2, Locale, BlockIndex | ||
+ | Store the above in a C# HashTable so we can easily access it | ||
+ | Advance Byte Buffer to beginning of BlockTable | ||
+ | Read next (BlockTableSize * 16) Bytes | ||
+ | Decrypt BlockTable | ||
+ | Read the following over and over (16 bytes total) from the decrypted stream: FileOffset, (compute FilePos), Compressed Size, FileSize, Flags | ||
+ | Store the above in a C# List so we can easily access it | ||
+ | </ | ||
+ | |||
+ | In order for the Hashtable to be super optimized the filename is converted to a number (hash) which is then stored. This hash contains information about which blockindex the block data resides in. The blockindex then tells us where the data for the filename is in the file. It's also important to note that there are apparently several different Hash Types. | ||
+ | |||
+ | From what I can determine: | ||
+ | ^ Hash Type ^ Purpose | ||
+ | | 0x000 | Hashes an Index | | ||
+ | | 0x100 | Hashes Name1 | | ||
+ | | 0x200 | Hashes Name2 | | ||
+ | | 0x300 | Hashes Table Data | | ||
+ | |||
+ | I'm not 100% sure where these Hash Types factor into the encryption. There appears to be two seeds and then the Hash Type is the offset for the hash. I don't know what this means. See SC2Inspector.MPQLogic.MPQUtilities.HashString(string input, int offset). | ||
+ | |||
+ | It seems like I done an incredible amount of work in just one day. I've been at this for over 12 hours it looks like. Oh well, I'm off to bed since it's 6am. I _REALLY_ hope I resume this project tomorrow. Not even going to close all my VS/Chrome windows. | ||
+ | |||
+ | Committed r2. | ||
+ | |||
+ | === 12/28/2011 @ 23:46 === | ||
+ | Ok, so I've gotten a decent amount done. I was looking at code for an MPQ parser and it looks like the SC2Replay files have a slightly different format. They start with MPQ\x1B then 1024 bytes into the file have another MPQ\x1A which actually starts the normal MPQ file. MPQ1B seems to be a StarCraft II only option for displaying additional metadata without having to read the file. | ||
+ | |||
+ | Using the following data taken from a random SC2Replay: | ||
+ | |||
+ | {{: | ||
+ | |||
+ | I was able to determine the following: [[sc2inspector# | ||
VLF represents a " | VLF represents a " | ||
Line 46: | Line 294: | ||
> | > | ||
>Source: http:// | >Source: http:// | ||
+ | |||
+ | I've taken the following code from a C# SC2Replay client to do this VLF for me: | ||
+ | <code csharp> | ||
+ | private static int ParseVLFNumber(BinaryReader reader) { | ||
+ | var bytes = 0; | ||
+ | var first = true; | ||
+ | var number = 0; | ||
+ | var multiplier = 1; | ||
+ | while (true) { | ||
+ | var i = reader.ReadByte(); | ||
+ | number += (i & 0x7F) * (int)Math.Pow(2, | ||
+ | if (first) { | ||
+ | if ((number & 1) != 0) { | ||
+ | multiplier = -1; | ||
+ | number--; | ||
+ | } | ||
+ | first = false; | ||
+ | } | ||
+ | if ((i & 0x80) == 0) { | ||
+ | break; | ||
+ | } | ||
+ | bytes++; | ||
+ | } | ||
+ | return (number / 2) * multiplier; | ||
+ | } | ||
+ | </ | ||
This took almost five hours to decipher with a LOT of help from various sources around the internet. I can't find it anymore but I thought I read somewhere that the game length was supposed to be in the header, but this could be incorrect. I would think the game length would be with the game recording date, players, colors, etc. | This took almost five hours to decipher with a LOT of help from various sources around the internet. I can't find it anymore but I thought I read somewhere that the game length was supposed to be in the header, but this could be incorrect. I would think the game length would be with the game recording date, players, colors, etc. | ||
Line 53: | Line 327: | ||
=== 12/28/2011 @ 18:31 === | === 12/28/2011 @ 18:31 === | ||
Decided to start on this project. I've added the ViewModelBase and made some changes to App.xaml and App.xaml.cs. Committed r1. | Decided to start on this project. I've added the ViewModelBase and made some changes to App.xaml and App.xaml.cs. Committed r1. | ||
+ | ===== Resources ===== | ||
+ | * SC2ReplayFormat Google Code (More Up To Date Than Above) - http:// | ||
+ | * MoPaQ Archive Format - http:// | ||
+ | * Mangos-Zero MQP Information - https:// | ||
+ | * PHPSC2Replay - http:// | ||
+ | * SC2Replay-CSharp - https:// | ||
+ | * Parsing SC2 Replays - http:// | ||
+ | * SC2Reader - https:// |