It is currently 14 Jun 2025, 19:02
   
Text Size

Information pending...

Moderator: CCGHQ Admins

Re: Information pending...

Postby thefiremind » 21 Nov 2014, 23:13

If someone wants to try spirolone's "decryption", here's the C# code which applies it to the whole file:
Code: Select all
byte[] source = File.ReadAllBytes(@"C:\<your_dotp2015_path>\DATA_000.ZED");
List<byte> destList = new List<byte>();
byte prev = 0;
for (int i=0; i<source.Length; i++)
{
    byte curr = source[i];
    curr ^= prev;
    destList.Add(curr);
    prev = source[i];
}
File.WriteAllBytes(@".\TEST_000.BIN", destList.ToArray<byte>());
Even in the expansion file, the end seems to be a PKZip central directory. Unfortunately, the rest of the file still makes no sense. Even a password-protected ZIP file should have a recognizable header.

EDIT: I just looked at the source code from the old Gibbed Tools and noticed that Pack and Unpack already use "Zlib" for compressed WAD files. WAD files have a recognizable header even when their content is compressed, though, so I don't know what to think about ZED files. I'm pretty sure that we just need to find where the actual zipped data starts, and figure out why only the central directory becomes readable after spirolone's "decryption". I would expect to find the file headers somewhere, which start with "50 4B 03 04", rather than "50 4B 01 02" as we see in the central directory.

EDIT 2: I wanted to take a closer look at a couple of old WADs, but I can't find traces of a central directory, even though they should be zipped as well. Not even after applying spirolone's "decryption". Are we missing something obvious?
Did someone else try and understand more from the old Gibbed Tools? To be honest I have never had to manage binary files, so it's difficult for me to follow the code flow, maybe someone else is more used to it.
< Former DotP 2012/2013/2014 modder >
Currently busy with life...
User avatar
thefiremind
Programmer
 
Posts: 3515
Joined: 07 Nov 2011, 10:55
Has thanked: 118 times
Been thanked: 722 times

Re: Information pending...

Postby NeoAnderson » 22 Nov 2014, 14:47

I just made some few tests with to simple XML cards files.
I zipped 2 cards into 2 zipped file ( Jalira, Master Polymorphist and Cancel ) then i zipped the 2 zip cards into one zip archive.
If we Hex Edit Cancel Zip we can see that the file starts with
Code: Select all
PKݾcEŽ5µÕ^2 NEO_M15_274_CANCEL_915383161.XMLµXysGÿ›­ÚïðÖAU$ã$ìAɤy ³È’W³@*¥jI-i`4£ÌÁáÔnI²
HEX :
Code: Select all
50 4B 03 04 14 00 00 00 08 00 DD BE 63 45 8E 35
B5 D5 5E 08 00 00 32 15 00 00 20 00 00 00 4E 45
4F 5F 4D 31 35 5F 32 37 34 5F 43 41 4E 43 45 4C
5F 39 31 35 33 38 33 31 36 31 2E 58 4D 4C B5 58
79 73 13 47 16 FF 9B AD DA EF F0 D6 7F 41 55 24
E3 24 EC 41 C9 A4 06 79 0C B3 C8 92 57 1A B3 40
2A A5 6A 49 2D 69 60 34 A3 CC C1 E1 D4 6E 49 B2
At end of the file we can see :
Code: Select all
ü-Ocõsè¾ÿPKݾcEŽ5µÕ^2 $ NEO_M15_274_CANCEL_915383161.XML
 á†Ñ1¹÷ÏÑ‘ä˜]ÐÑ‘ä˜]ÐPKrœ
HEX :
Code: Select all
FC 2D 15 10 4F 63 F5 73 E8 BE FF 02 50 4B 01 02
1F 00 14 00 00 00 08 00 DD BE 63 45 8E 35 B5 D5
5E 08 00 00 32 15 00 00 20 00 24 00 00 00 00 00
00 00 20 00 00 00 00 00 00 00 4E 45 4F 5F 4D 31
35 5F 32 37 34 5F 43 41 4E 43 45 4C 5F 39 31 35
33 38 33 31 36 31 2E 58 4D 4C 0A 00 20 00 00 00
00 00 01 00 18 00 E1 86 D1 31 B9 F7 CF 01 D1 91
E4 98 5D 06 D0 01 D1 91 E4 98 5D 06 D0 01 50 4B
05 06 00 00 00 00 01 00 01 00 72 00 00 00 9C 08
00 00 00 00
----------------------------------------------------------------------



Now if we edit the Zip archive with the 2 zipped files we cans see :
Code: Select all
PKíxvEçk×j   $    NEO_M15_274_CANCEL_915383161.zipVy8Œ¹Ms_aZÚäÃt8Ê¡a®Js”¡œ+"eF’b‘+±äV®åÓÁ'b&ÌB„˜ÖäÈùõ=ßïùý
HEX :
Code: Select all
50 4B 03 04 14 00 00 00 08 00 ED 78 76 45 E7 6B
D7 6A 16 09 00 00 24 09 00 00 20 00 00 00 4E 45
4F 5F 4D 31 35 5F 32 37 34 5F 43 41 4E 43 45 4C
5F 39 31 35 33 38 33 31 36 31 2E 7A 69 70 8D 56
79 38 13 8C 03 1E B9 4D 73 5F 61 5A DA E4 18 C3
74 38 CA 1D A1 61 AE 4A 73 94 A1 9C 2B 22 1F 65
46 92 62 91 19 2B B1 E4 56 AE E5 1B D3 C1 27 62
26 CC 11 42 84 98 D6 E4 C8 F9 F5 3D DF EF F9 FD
After the first data block we can find another 50 4B 03 04 header :
Code: Select all
ä÷7PKçxvEaŵë4NEO_M15_064_JALIRA_MASTER_POLYMORPHIST_915383287.zipWç;£©­µgÕ*¥ªµ75"Ä&FcÇnìÕØ4BE(ÕÚ«b”šÕT­¢füJÑ f­ÚmµÞ÷
HEX :
Code: Select all
E4 1F F7 37 50 4B 03 04 14 00 00 00 08 00 E7 78
76 45 61 C5 17 B5 EB 0E 00 00 0D 0F 00 00 34 00
00 00 4E 45 4F 5F 4D 31 35 5F 30 36 34 5F 4A 41
4C 49 52 41 5F 4D 41 53 54 45 52 5F 50 4F 4C 59
4D 4F 52 50 48 49 53 54 5F 39 31 35 33 38 33 32
38 37 2E 7A 69 70 9D 57 E7 3B 1B 0E A3 0D A9 AD
B5 67 D5 2A A5 AA B5 37 35 22 C4 26 46 63 C7 6E
EC D5 D8 34 42 45 28 D5 DA AB 62 14 8D 11 94 9A
D5 54 AD A2 66 FC 4A D1 20 66 AD DA 6D B5 DE F7
At end of file we will find the two 50 4B 01 02 headers :
Code: Select all
PKíxvEçk×j   $    $ NEO_M15_274_CANCEL_915383161.zip U1¥]ÐØ/¥]ÐØ/¥]ÐPKçxvEaŵë
4$ T   NEO_M15_064_JALIRA_MASTER_POLYMORPHIST_915383287.zip síó]Ðûoò]Ðûoò]ÐPKø‘
HEX :
Code: Select all
01 50 4B 01 02 1F 00 14 00 00 00 08 00 ED 78 76
45 E7 6B D7 6A 16 09 00 00 24 09 00 00 20 00 24
00 00 00 00 00 00 00 20 00 00 00 00 00 00 00 4E
45 4F 5F 4D 31 35 5F 32 37 34 5F 43 41 4E 43 45
4C 5F 39 31 35 33 38 33 31 36 31 2E 7A 69 70 0A
00 20 00 00 00 00 00 01 00 18 00 7F 55 31 A5 5D
06 D0 01 07 D8 2F A5 5D 06 D0 01 07 D8 2F A5 5D
06 D0 01 50 4B 01 02 1F 00 14 00 00 00 08 00 E7
78 76 45 61 C5 17 B5 EB 0E 00 00 0D 0F 00 00 34
00 24 00 00 00 00 00 00 00 20 00 00 00 54 09 00
00 4E 45 4F 5F 4D 31 35 5F 30 36 34 5F 4A 41 4C
49 52 41 5F 4D 41 53 54 45 52 5F 50 4F 4C 59 4D
4F 52 50 48 49 53 54 5F 39 31 35 33 38 33 32 38
37 2E 7A 69 70 0A 00 20 00 00 00 00 00 01 00 18
00 73 ED F3 9D 5D 06 D0 01 FB 6F F2 9D 5D 06 D0
01 FB 6F F2 9D 5D 06 D0 01 50 4B 05 06 00 00 00
00 02 00 02 00 F8 00 00 00 91 18 00 00 00 00
As we can see from the examples above the Data headers starts with 50 4B 03 04 the files Descriptors headers starts with 50 4B 01 02 and the end of Zip archive has 50 4B 05 06.
The fact that you can only see the 50 4B 01 02 headers inside the ZED i think should means that the data is encrypted so also the File Data Headers.
NeoAnderson
 
Posts: 914
Joined: 10 Sep 2013, 07:49
Has thanked: 18 times
Been thanked: 139 times

Re: Information pending...

Postby thefiremind » 22 Nov 2014, 15:35

NeoAnderson wrote:The fact that you can only see the 50 4B 01 02 headers inside the ZED i think should means that the data is encrypted so also the File Data Headers.
But then why encrypting the most part of the file and just scrambling the central directory through XOR? That doesn't make sense to me... I would rather have encrypted the whole file. My guess is that the part of the file where spirolone's method doesn't work, received another kind of scrambling, as easy as that one, but of course different. It would also be more lightweight than "real" encryption performance-wise, when the engine needs to read it. At Stainless they probably like to play with bitwise operators: the profile hash is another example. :lol:

EDIT: Moreover, you can't find either of the PKZip signatures in the old WADs, yet Gibbed Tools uses Zlib when they contain compressed data, and I can't find any trace of encryption/decryption methods inside the source code. How do you explain that?
EDIT 2: I'll answer to myself: if I understood correctly, the functions used in Gibbed Tools read/write something that could be called "raw compressed data", that is without headers or additional information. So old WADs probably didn't need any PKZip header to work. If we assume that to be true for ZED as well, could it be that picking a piece of ZED file with starting point and length as instructed by the central directory and passing it through an InflaterInputStream as in Gibbed Tools is enough to obtain the original file?
< Former DotP 2012/2013/2014 modder >
Currently busy with life...
User avatar
thefiremind
Programmer
 
Posts: 3515
Joined: 07 Nov 2011, 10:55
Has thanked: 118 times
Been thanked: 722 times

Re: Information pending...

Postby NeoAnderson » 22 Nov 2014, 16:15

thefiremind wrote:
NeoAnderson wrote:The fact that you can only see the 50 4B 01 02 headers inside the ZED i think should means that the data is encrypted so also the File Data Headers.
But then why encrypting the most part of the file and just scrambling the central directory through XOR? That doesn't make sense to me... I would rather have encrypted the whole file. My guess is that the part of the file where spirolone's method doesn't work, received another kind of scrambling, as easy as that one, but of course different. It would also be more lightweight than "real" encryption performance-wise, when the engine needs to read it. At Stainless they probably like to play with bitwise operators: the profile hash is another example. :lol:

EDIT: Moreover, you can't find either of the PKZip signatures in the old WADs, yet Gibbed Tools uses Zlib when they contain compressed data, and I can't find any trace of encryption/decryption methods inside the source code. How do you explain that?
Fire honestly i don't know as like many other guys i am just trying to put in play some ideas, i am a "greenhorn" about Binary files and even more about encryption.
About the PK signature there is 2 calls inside DATA_CORE.WAD (dotp 2014) signature 50 4B 01 02.
About the Stainless approach i can agree with you that probably they just mess up the file using some different Math or logic operation onto binary, but our hope is just to test any crazy idea to get some results.

About the PK encryption i am still not sure because i have read the link i posted some post ago about Pkzip structure and there are some infos that still let me think about an ecrypted zip structure. I post here a short version with most relevant infos :


"The archive consists of a series of local file descriptors, each containing a local file header, the actual compressed and/or encrypted data, as well as an optional data descriptor. Whether a data descriptor exists or not depends on a flag in the local file header.
Following the file descriptors is the archive decryption header, which only exists in PKZip file version 6.2 or greater. This header is only present if the central directory is encrypted and contains information about the encryption specification. The archive extra data record is also only for file of version 6.2 or greater and is not present in all Zip files. It is used in to support the encryption or compression of the central directory.
The central directory summarizes the local file descriptors and carries additional information regarding file attributes, file comments, location of the local headers, and multi-file archive information. "
---------------
The signature of the local file header. This is always '\x50\x4b\x03\x04'.
--------------

Archive decryption header
This header is used to support the Central Directory Encryption Feature. It is present when the central directory is encrypted. The format of this data record is identical to the Decryption header record preceding compressed file data.

Archive extra data record
This header is used to support the Central Directory Encryption Feature. When present, this record immediately precedes the central directory data structure. The size of this data record will be included in the Size of the Central Directory field in the End of Central Directory record.

Central directory
The central directory contains more metadata about the files in the archive and also contains encryption information and information about Zip64 (64-bit zip archives) archives. Furthermore, the central directory contains information about archives that span multiple files
The file headers are similar to the local file headers, but contain some extra information. The Zip64 entries handle the case of a 64-bit Zip archive, and the end of the central directory record contains information about the archive itself
The signature of the file header. This is always '\x50\x4b\x01\x02'.
The signature of end of central directory record. This is always '\x50\x4b\x05\x06'.
NeoAnderson
 
Posts: 914
Joined: 10 Sep 2013, 07:49
Has thanked: 18 times
Been thanked: 139 times

Re: Information pending...

Postby RiiakShiNal » 22 Nov 2014, 16:42

thefiremind wrote:EDIT 2: I wanted to take a closer look at a couple of old WADs, but I can't find traces of a central directory, even though they should be zipped as well. Not even after applying spirolone's "decryption". Are we missing something obvious?
Did someone else try and understand more from the old Gibbed Tools? To be honest I have never had to manage binary files, so it's difficult for me to follow the code flow, maybe someone else is more used to it.
The code flow isn't that hard to follow (at least not for me), the central directory of the old WAD files is actually pretty easy to understand, it is just structured differently from the ZEDs. In the WADs we have our header (magic number, version, flags, length of XML header, XML header, length of strings, strings [individual strings null terminated], number of data types, list of data types [if any], file count, directory count, number of offsets, list of offsets, directory/file entry structure [4 ints per entry]). Basically all the strings are together and all the offsets are together. The directory/file entry structure determines the file hierarchy (as well as the lengths of the files at the appropriate offsets). The flags for the main WAD determine whether the contained files are zlib compressed or not.
RiiakShiNal
Programmer
 
Posts: 2188
Joined: 16 May 2011, 21:37
Has thanked: 75 times
Been thanked: 497 times

Re: Information pending...

Postby NeoAnderson » 22 Nov 2014, 19:06

thefiremind wrote:But then why encrypting the most part of the file and just scrambling the central directory through XOR? That doesn't make sense to me...
I was just thinking that probably they have obfuscated only the headers section with xor function, just because was the only readable part, so they would hide the visible infos.
As before this is just a supposition.
NeoAnderson
 
Posts: 914
Joined: 10 Sep 2013, 07:49
Has thanked: 18 times
Been thanked: 139 times

Re: Information pending...

Postby thefiremind » 23 Nov 2014, 14:56

I made a test program that reads the integers before and after a filename inside the central directory of a ZED file.

Here's the source:
ZedTest source code (Program.cs of a C# console application) | Open
Code: Select all
using System;
using System.IO;
using System.Text;

namespace ZedTest
{
    class Program
    {
        private static int GetFinalOffset(Stream stream)
        {
            byte[] last8 = new byte[8];
            stream.Seek(-8L, SeekOrigin.End);
            stream.Read(last8, 0, 8);
            byte[] offsetBytes = new byte[4];
            for (int i = 0; i < 4; i++)
                offsetBytes[i] = Convert.ToByte(last8[i * 2] ^ last8[i * 2 + 1]);
            return BitConverter.ToInt32(offsetBytes, 0);
        }

        private static void ProcessFile(string path, string internalFile)
        {
            if (!File.Exists(path))
                Console.WriteLine(path + "not found");
            else
                using (FileStream input = File.OpenRead(path))
                {
                    int finalOffset = GetFinalOffset(input);
                    Console.WriteLine("Final offset for {0}: {1}", path, finalOffset);
                    input.Seek(finalOffset - 1, SeekOrigin.Begin);
                    int dirLength = Convert.ToInt32(input.Length) - finalOffset - 8;
                    byte[] dirScrambled = new byte[dirLength + 1];
                    input.Read(dirScrambled, 0, dirLength + 1);
                    byte[] dir = new byte[dirLength];
                    for (int i = 0; i < dirLength; i++)
                        dir[i] = Convert.ToByte(dirScrambled[i + 1] ^ dirScrambled[i]);
                    string dirString = Encoding.ASCII.GetString(dir);
                    int offset;
                    int internalFileOffset = dirString.IndexOf(internalFile);
                    if (internalFileOffset == -1)
                        Console.WriteLine(internalFile + "not found");
                    else
                    {
                        Console.WriteLine("Integers around {0} in {1}:", internalFile, path);
                        for (int i = -9; i < 0; i++)
                        {
                            offset = internalFileOffset + i * 4;
                            Console.WriteLine("* Offset: {0}, Value: {1}", offset, BitConverter.ToUInt32(dir, offset));
                        }
                        Console.WriteLine("* > Offset: {0}, Value: {1}", internalFileOffset, internalFile);
                        for (int i = 0; i < 9; i++)
                        {
                            offset = internalFileOffset + internalFile.Length + i * 4;
                            Console.WriteLine("* Offset: {0}, Value: {1}", offset, BitConverter.ToUInt32(dir, offset));
                        }
                    }
                }
            Console.WriteLine("END" + System.Environment.NewLine);
        }

        static void Main(string[] args)
        {
            string dotpDir = @"C:\Games\Magic 2015";
            string headerName = ".METADATA/Header.xml";

            ProcessFile(Path.Combine(dotpDir, "DATA_000.ZED"), headerName);
            ProcessFile(Path.Combine(dotpDir, "MOVIES_000.ZED"), headerName);
            ProcessFile(Path.Combine(dotpDir, "AUDIO_000.ZED"), headerName);
            ProcessFile(Path.Combine(dotpDir, "DATA_010.ZED"), headerName);
            ProcessFile(Path.Combine(dotpDir, "MOVIES_010.ZED"), headerName);
            ProcessFile(Path.Combine(dotpDir, "AUDIO_010.ZED"), headerName);

            Console.Write("Press a key to close...");
            Console.ReadKey();
        }
    }
}
Here's the output:
ZedTest output | Open
Final offset for C:\Games\Magic 2015\DATA_000.ZED: 382409068
Integers around .METADATA/Header.xml in C:\Games\Magic 2015\DATA_000.ZED:
* Offset: 248, Value: 1975279334
* Offset: 252, Value: 1100305658
* Offset: 256, Value: 33963293
* Offset: 260, Value: 179306496
* Offset: 264, Value: 1310720
* Offset: 268, Value: 36
* Offset: 272, Value: 0
* Offset: 276, Value: 32
* Offset: 280, Value: 65524
* > Offset: 284, Value: .METADATA/Header.xml
* Offset: 304, Value: 2097162
* Offset: 308, Value: 0
* Offset: 312, Value: 1572865
* Offset: 316, Value: 2844720013
* Offset: 320, Value: 30405948
* Offset: 324, Value: 785907062
* Offset: 328, Value: 30405950
* Offset: 332, Value: 785907062
* Offset: 336, Value: 30405950
END

Final offset for C:\Games\Magic 2015\MOVIES_000.ZED: 237151478
Integers around .METADATA/Header.xml in C:\Games\Magic 2015\MOVIES_000.ZED:
* Offset: 248, Value: 3484301348
* Offset: 252, Value: 18321603
* Offset: 256, Value: 16165377
* Offset: 260, Value: 16121856
* Offset: 264, Value: 1310720
* Offset: 268, Value: 36
* Offset: 272, Value: 0
* Offset: 276, Value: 32
* Offset: 280, Value: 1765
* > Offset: 284, Value: .METADATA/Header.xml
* Offset: 304, Value: 2097162
* Offset: 308, Value: 0
* Offset: 312, Value: 1572865
* Offset: 316, Value: 650273722
* Offset: 320, Value: 30405949
* Offset: 324, Value: 841382386
* Offset: 328, Value: 30405951
* Offset: 332, Value: 841382386
* Offset: 336, Value: 30405951
END

Final offset for C:\Games\Magic 2015\AUDIO_000.ZED: 376478166
Integers around .METADATA/Header.xml in C:\Games\Magic 2015\AUDIO_000.ZED:
* Offset: 248, Value: 2192550519
* Offset: 252, Value: 2478077023
* Offset: 256, Value: 14950445
* Offset: 260, Value: 14942208
* Offset: 264, Value: 1310720
* Offset: 268, Value: 36
* Offset: 272, Value: 0
* Offset: 276, Value: 32
* Offset: 280, Value: 155026
* > Offset: 284, Value: .METADATA/Header.xml
* Offset: 304, Value: 2097162
* Offset: 308, Value: 0
* Offset: 312, Value: 1572865
* Offset: 316, Value: 3272533153
* Offset: 320, Value: 30405946
* Offset: 324, Value: 642400517
* Offset: 328, Value: 30405951
* Offset: 332, Value: 642400517
* Offset: 336, Value: 30405951
END

Final offset for C:\Games\Magic 2015\DATA_010.ZED: 45491994
Integers around .METADATA/Header.xml in C:\Games\Magic 2015\DATA_010.ZED:
* Offset: 248, Value: 2844751646
* Offset: 252, Value: 3507365508
* Offset: 256, Value: 32280999
* Offset: 260, Value: 86441984
* Offset: 264, Value: 1310720
* Offset: 268, Value: 36
* Offset: 272, Value: 0
* Offset: 276, Value: 32
* Offset: 280, Value: 7969
* > Offset: 284, Value: .METADATA/Header.xml
* Offset: 304, Value: 2097162
* Offset: 308, Value: 0
* Offset: 312, Value: 1572865
* Offset: 316, Value: 3352679261
* Offset: 320, Value: 30405948
* Offset: 324, Value: 1777445983
* Offset: 328, Value: 30405951
* Offset: 332, Value: 1777445983
* Offset: 336, Value: 30405951
END

Final offset for C:\Games\Magic 2015\MOVIES_010.ZED: 161
Integers around .METADATA/Header.xml in C:\Games\Magic 2015\MOVIES_010.ZED:
* Offset: 248, Value: 3185762384
* Offset: 252, Value: 3677816110
* Offset: 256, Value: 9443295
* Offset: 260, Value: 9437184
* Offset: 264, Value: 1310720
* Offset: 268, Value: 36
* Offset: 272, Value: 0
* Offset: 276, Value: 32
* Offset: 280, Value: 17
* > Offset: 284, Value: .METADATA/Header.xml
* Offset: 304, Value: 2097162
* Offset: 308, Value: 0
* Offset: 312, Value: 1572865
* Offset: 316, Value: 653901908
* Offset: 320, Value: 30405949
* Offset: 324, Value: 2260058185
* Offset: 328, Value: 30405951
* Offset: 332, Value: 2260058185
* Offset: 336, Value: 30405951
END

Final offset for C:\Games\Magic 2015\AUDIO_010.ZED: 19570282
Integers around .METADATA/Header.xml in C:\Games\Magic 2015\AUDIO_010.ZED:
* Offset: 248, Value: 1766988382
* Offset: 252, Value: 905356580
* Offset: 256, Value: 14952491
* Offset: 260, Value: 14942208
* Offset: 264, Value: 1310720
* Offset: 268, Value: 36
* Offset: 272, Value: 0
* Offset: 276, Value: 32
* Offset: 280, Value: 4994
* > Offset: 284, Value: .METADATA/Header.xml
* Offset: 304, Value: 2097162
* Offset: 308, Value: 0
* Offset: 312, Value: 1572865
* Offset: 316, Value: 3866086882
* Offset: 320, Value: 30405946
* Offset: 324, Value: 2250007180
* Offset: 328, Value: 30405951
* Offset: 332, Value: 2250007180
* Offset: 336, Value: 30405951
END

Press a key to close...
MOVIES_010.ZED is quite informative because it actually contains nothing else than "<Files> </Files>", the XML header, and the xor-scrambled part. It's easy to verify that the integer immediately before the filename (whose value is 17 for MOVIES_010) really is the offset of the file data (in MOVIES_010 the XML header starts just after "<Files> </Files>" which is 16 characters long). One thing, though: where is the file length? How can the engine read the file packed into the ZED (be it encrypted, compressed or whatever) if it doesn't know how long it is?

Since spirolone only mentioned 1 integer before the filename, there's no particular reason why I decided to try and read 9 integers (other than symmetry with the 9 integers after the filename).

I'm attaching MOVIES_010.ZED, for those who don't have the expansion, in case someone else wants to take a look at it.
Attachments
MOVIES_010.zip
A basically empty ZED file
(647 Bytes) Downloaded 534 times
< Former DotP 2012/2013/2014 modder >
Currently busy with life...
User avatar
thefiremind
Programmer
 
Posts: 3515
Joined: 07 Nov 2011, 10:55
Has thanked: 118 times
Been thanked: 722 times

Re: Information pending...

Postby NeoAnderson » 23 Nov 2014, 17:23

thefiremind wrote:One thing, though: where is the file length? How can the engine read the file packed into the ZED (be it encrypted, compressed or whatever) if it doesn't know how long it is?
Maybe they read with a loop until some certain values. I checked the output values of your test, to better compare i have copied them it into an Excel worksheet, and i can see that there are some fixed values for certain offsets, and also some incremental values for other offsets.
Image
NeoAnderson
 
Posts: 914
Joined: 10 Sep 2013, 07:49
Has thanked: 18 times
Been thanked: 139 times

Re: Information pending...

Postby thefiremind » 23 Nov 2014, 17:37

Nice, but the length isn't there: the length of the XML header in MOVIES_010 should be 144 (first byte is at 17 and last byte is at 160) and I can't see it. It's totally possible that some bytes were misinterpreted: I considered all of them as 32-bit unsigned integers, but who knows if that's really how they should be read.
< Former DotP 2012/2013/2014 modder >
Currently busy with life...
User avatar
thefiremind
Programmer
 
Posts: 3515
Joined: 07 Nov 2011, 10:55
Has thanked: 118 times
Been thanked: 722 times

Re: Information pending...

Postby NeoAnderson » 23 Nov 2014, 17:44

thefiremind wrote:Nice, but the length isn't there: the length of the XML header in MOVIES_010 should be 144 (first byte is at 17 and last byte is at 160) and I can't see it. It's totally possible that some bytes were misinterpreted: I considered all of them as 32-bit unsigned integers, but who knows if that's really how they should be read.
I know i have copied them into a worksheet only to have simultaneous view of all of them. About the values i have picked them from your previous post. About the data type i understand that most of them could be Integer, LongIntegers, Single, Double, Byte, ....
About MOVIES_010 if you subtract 161 (header value) - 17(offset 280) you obtain = 144
NeoAnderson
 
Posts: 914
Joined: 10 Sep 2013, 07:49
Has thanked: 18 times
Been thanked: 139 times

Re: Information pending...

Postby thefiremind » 23 Nov 2014, 19:41

NeoAnderson wrote:About MOVIES_010 if you subtract 161 (header value) - 17(offset 280) you obtain = 144
That's because there's nothing else in that ZED file, so the central directory starts where the first and only file ends. It is possible that the length is calculated just by looking at where the next file starts (or the central directory if it's the last file), but it seems a bit unusual to me.

Anyway, I tried to open DATA_010, start from offset 7969, pick various chunk lengths from 100 to 500 bytes, and pass them through InflateInputStream. Zlib has never been able to find something readable. Either I did something wrong in the procedure, or there's still some unscrambling to do, as I suspected before (or there's some kind of encryption, but as I said, I find it less likely).
< Former DotP 2012/2013/2014 modder >
Currently busy with life...
User avatar
thefiremind
Programmer
 
Posts: 3515
Joined: 07 Nov 2011, 10:55
Has thanked: 118 times
Been thanked: 722 times

Re: Information pending...

Postby NeoAnderson » 23 Nov 2014, 22:38

The chunk size shouldn't be the issue, because it is just a buffer, and i read that it can be used any size to decompress, standard size are 8k or 16k.
The fact that you didn't find anything is a signal that the data is still scrambled.

Update: if you look at previous page where i posted about dotp Exe file, the section about inflate function you can see that is used SHA-1RSA encryption and a Montgomery reduction algorythm to improve the performance.
NeoAnderson
 
Posts: 914
Joined: 10 Sep 2013, 07:49
Has thanked: 18 times
Been thanked: 139 times

Re: Information pending...

Postby RiiakShiNal » 24 Nov 2014, 00:07

About the size of the file block in the the ZED it should be relatively easy to get if we have the the central directory and all of the beginning offsets as then we just take then next offset and subtract the previous offset. If it is the last file in the ZED we take the offset of the start of the central dictionary and subtract the offset of that final file to get the size. This is fairly common for some applications.

SHA-1 is a message hashing algorithm used for signatures to make sure the message hasn't been tampered with (in this case that the file hasn't been tampered with). It is a one-way algorithm and you can't get the original message from a SHA-1 hash (which is always the same size regardless if it was run on a 1KB file or a 1GB file).
RiiakShiNal
Programmer
 
Posts: 2188
Joined: 16 May 2011, 21:37
Has thanked: 75 times
Been thanked: 497 times

Re: Information pending...

Postby NeoAnderson » 24 Nov 2014, 00:13

I was looking for SHA-1RSA decrypter and i found some website that could use enourmous dictonary (43.745 billion unique decrypted SHA1 hashes) to recover the SHA password, but we need the hash of the file..someone knows how to retrieve it?
If someone knows or want to take a look here is the site : HashKiller.CO.UK
RiiakShiNal wrote:SHA-1 is a message hashing algorithm used for signatures to make sure the message hasn't been tampered with (in this case that the file hasn't been tampered with). It is a one-way algorithm and you can't get the original message from a SHA-1 hash (which is always the same size regardless if it was run on a 1KB file or a 1GB file).
So do you also think the data is not using any Encryption? why there are references to private key, RSA,....???
NeoAnderson
 
Posts: 914
Joined: 10 Sep 2013, 07:49
Has thanked: 18 times
Been thanked: 139 times

Re: Information pending...

Postby spirolone » 24 Nov 2014, 00:20

Maybe I have some infos usefull. I'm pretty sure that structure is same that PKZip central directory one ( https://users.cs.jmu.edu/buchhofp/foren ... pkzip.html ) and I attach an Excel sheet with values for a Data_000.zed file. As you can see there are all infos usefull about single files in archive (you didn't see compressed size cause there is a 2-bytes integer with lenght of file name). In a standard zip file we should have an "End of central directory record" and a local file header at the beginning of any file, but there are only redundant data and we can easily construct them. I tried to do so, but it was not sufficient. Now, I'm doing some tests ad then I'll update you...
Attachments
Header.zip
(751.76 KiB) Downloaded 679 times
spirolone
Programmer
 
Posts: 190
Joined: 31 Aug 2014, 23:14
Has thanked: 7 times
Been thanked: 107 times

PreviousNext

Return to 2015

Who is online

Users browsing this forum: No registered users and 1 guest


Who is online

In total there is 1 user online :: 0 registered, 0 hidden and 1 guest (based on users active over the past 10 minutes)
Most users ever online was 4143 on 23 Jan 2024, 08:21

Users browsing this forum: No registered users and 1 guest

Login Form