MagosX.com

General Forums => File Distribution => Topic started by: Magos on February 11, 2009, 04:19:24 PM



Title: MpqLib
Post by: Magos on February 11, 2009, 04:19:24 PM
MpqLib v1.00

MdxLib is a .NET 3.5 Class Library to manipulate MPQ archives. You can open or create archives, then open, read, write, delete, import, export files from them. It's a wrapper over StormLib (http://www.zezula.net/en/mpq/stormlib.html) to make usage easier. It's mainly aimed for programmers to be used in tools and similar. It was written in C++.NET, but can be used by any of the .NET languages (and perhaps by other languages if you have some skillz).

  • An object oriented approach to manipulating MPQ archives.
  • Open or create MPQ archives.
  • Open, read, write, delete, import, export files in MPQ archives.

Download it here:
MpqLib.zip (http://files.magosx.com/download.php?file=MpqLib.zip)

Source code (released under GPL):
MpqLibSource.zip (http://files.magosx.com/download.php?file=MpqLibSource.zip)


Title: Re: MpqLib
Post by: Magos on February 11, 2009, 04:20:39 PM
Example 1 - Export a file

This first simple example shows the simplicity of using the library. First you need an archive object pointing to the main WarCraft 3 MPQ (War3.mpq). After the archive has been opened (an exception is thrown if an error occurs, like if the archive cannot be found) you can call the method ExportFile to export a file from the archive.

I'm using the "using"-directive on the archive which means it will be automatically closed and disposed when done (even if an exception is thrown). If you prefer you can call Close or Dispose manually.

Code:
//
// Opens the main WarCraft 3 MPQ archive.
//
using(var Archive = new MpqLib.Mpq.CArchive("C:\\Program Files\\Warcraft III\\War3.mpq"))
{
//
// Exports a file from the archive. The first path refers to the path inside the archive.
// The second path refers to where the file should be exported.
//
Archive.ExportFile("Sound\\Music\\mp3Music\\Comradeship.mp3", "C:\\Temp\\Comradeship.mp3");
}

Example 2 - Import a file

This next example imports a file (the file we just exported). Note the new second argument to the archive constructor, it creates a new archive if none exists (if false or left out it will throw an exception if no archive exists).

The import syntax is basically the same as the export syntax, only you have an extra argument telling which compression to use. Not all compressions may work in all cases. Implode is a good general purpose compression though, so we use it.

Finally we open a stream to the file we just imported. We don't read any data, just retrieve its filesize and display it to the user.

Code:
//
// Creates a new MPQ archive. The second parameter decides if a new
// archive should be created (if none exists). If you always want
// to create a new, use System.IO.File.Delete first.
//
using(var Archive = new MpqLib.Mpq.CArchive("C:\\Temp\\Temp.mpq", true))
{
//
// Imports a file into the archive, compressing it using the Implode algorithm
// (Mp3 files are already compressed so the size won't be reduced much).
//
Archive.ImportFile("MyOwnFolder\\Comradeship.mp3", "C:\\Temp\\Comradeship.mp3", MpqLib.Mpq.ECompression.Implode);

//
// Streams the newly imported file, displaying its real size and
// the size it has in the archive (usually differs if compressed).
//
using(var File = new MpqLib.Mpq.CFileStream(Archive, "MyOwnFolder\\Comradeship.mp3"))
{
System.Windows.Forms.MessageBox.Show("Size: " + File.Length + System.Environment.NewLine + "Compressed Size: " + File.CompressedLength);
}
}

Example 3 - Combining MdxLib and MpqLib

In this example we'll open an MDX model with MpqLib then retrieve some information from it using MdxLib. Since the MPQ file stream implements the standard .NET stream functionality it can be used not only in MdxLib but in any function that accepts standard streams.

Code:
//
// Opens an MPQ archive.
//
using(var Archive = new MpqLib.Mpq.CArchive("C:\\Program Files\\Warcraft III\\War3.mpq"))
{
//
// Opens an MDX model.
//
using(var File = new MpqLib.Mpq.CFileStream(Archive, "Units\\Human\\Footman\\Footman.mdx"))
{
//
// Creates a new model and an MDX format object (for loading MDX models).
//
var Model = new MdxLib.Model.CModel();
var MdxFormat = new MdxLib.ModelFormats.CMdx();

//
// Loads the MDX model.
//
MdxFormat.Load("Footman.mdx", File, Model);

//
// Displays the model name and how many textures it has.
//
System.Windows.Forms.MessageBox.Show(Model.Name +  " has " + Model.Textures.Count + " textures!");
}
}


Title: Re: MpqLib
Post by: Magos on August 24, 2009, 09:35:34 AM
Source code released (under GPL). See the first post.


Title: Re: MpqLib
Post by: pha0001 on December 25, 2009, 04:20:29 AM
Hello Magos,

I was wondering if MpqLib can possibly find files without the internal or external listfile (that is, finding files on it's own). I was wondering if it could do this or if possible be implemented, or otherwise advice. Thanks.


Title: Re: MpqLib
Post by: Magos on December 26, 2009, 05:34:04 AM
Nope, not possible. No filename information is stored inside the mpq, only hashed values. You have to know the name you're looking for, thus the need for a filelist.


Title: Re: MpqLib
Post by: pha0001 on December 26, 2009, 05:41:59 AM
Nope, not possible. No filename information is stored inside the mpq, only hashed values. You have to know the name you're looking for, thus the need for a filelist.

Hmm... Thanks for replying, but I was wondering if it is possible to find the filename in the hashed values with the use of Storm.dll or some sort like that. Thanks again.


Title: Re: MpqLib
Post by: Dege on December 26, 2009, 09:57:16 AM
i have another question: how can i delete the (attributes) file in a warcraft 3 map?

i've tried but don't work...


Title: Re: MpqLib
Post by: Magos on December 27, 2009, 03:35:27 PM
If

Code:
RemoveFile("(attributes)");

fails then there's probably a limitation in stormlib. Not much to do.


Title: Re: MpqLib
Post by: pha0001 on December 27, 2009, 08:43:21 PM
If

Code:
RemoveFile("(attributes)");

fails then there's probably a limitation in stormlib. Not much to do.

I guess, but with ShadowFlare's MPQ assembly it works quite well deleting the listfile and the attributes file too, from what I remember last time.


Title: Re: MpqLib
Post by: sikele2236 on August 15, 2010, 09:03:59 PM
Code:


 ........         
  if (mpq.FileExists("scripts\\war3map.j"))
                jpath = "scripts\\war3map.j";

            mpq.ExportFile(jpath, "war3map.j"); // succeed
            mpq.RemoveFile(jpath);
            byte[] bytej = File.ReadAllBytes("war3map.j");
            //do something
            mpq.ImportFile(jpath, utf8.GetBytes(jstring));
            mpq.ExportFile(jpath, "war3map.j"); // succeed
            mpq.RemoveFile(jpath);
            mpq.ImportFile(jpath, utf8.GetBytes(jstring), MpqLib.Mpq.ECompression.ZLib);
            mpq.ExportFile(jpath, "war3map.j"); // unable to export.......



Title: Re: MpqLib
Post by: vercas on August 15, 2010, 10:37:48 PM
Magos, I have a retarded exception comming for no reason from ..ctor! (with double dots!)
Please pm me or reply my PM!


Title: Re: MpqLib
Post by: Magos on August 16, 2010, 09:52:27 AM
Can you post the call that fails and the exception that you get?


Title: Re: MpqLib
Post by: qcz on August 16, 2010, 02:00:02 PM
If I import files to archives using MpqLib with Zlib/BZip2 compression, later I won't be able to read that file from the archive.
(Tried to export after import with MpqLib and with Mpq Editor)

Code:
mpqfile.ImportFile(backupMpqFileName, realFileName, ECompression.ZLib);
mpqfile.ExportFile(backupMpqFileName, outFileName);  // won't work


Title: Re: MpqLib
Post by: sikele2236 on August 16, 2010, 07:03:52 PM
Code:
mpq.ImportFile(jpath, utf8.GetBytes(jstring), MpqLib.Mpq.ECompression.ZLib);
mpq.ExportFile(jpath, "war3map.j"); // unable to export.......

未处理 System.IO.IOException
  Message="Unable to export \"scripts\\war3map.j\" as \"war3map.j\"!"
  Source="MpqLib"
  StackTrace:
       在 MpqLib.Mpq.CArchive.ExportFile(String FileName, String RealFileName)


Title: Re: MpqLib
Post by: Magos on August 17, 2010, 09:13:26 AM
Try other compression methods than ZLib (or ignore that parameter and use the default value).
Try specifying the full path (like C:\Folder\File.ext) in the filename.


Title: Re: MpqLib
Post by: sikele2236 on August 17, 2010, 09:37:35 AM
Code:
mpq.ImportFile(jpath, utf8.GetBytes(crtJfile), MpqLib.Mpq.ECompression.Implode);
 mpq.ExportFile(jpath, "c:\\war3map.j");

unable to .......:
huffman,Bzip2,Zlib,
succeed:
Implode


Title: Re: MpqLib
Post by: qcz on August 22, 2010, 06:38:26 AM
Hi again!
Would you make a new version with the latest StormLib? Or give a brief description how to compile it in VS2008/10 (where I have to put StormLib files, what to do w/ them)? I want to access files from StarCraft II, but it's using LZMA compression, and it's introduced in later versions of StormLib.

I spent an hour trying to compile, fix things, but as I never programmed in C++, not succeeded.


Title: Re: MpqLib
Post by: Magos on August 24, 2010, 09:04:01 AM
It's been a long time, but from what i remember you compile the stormlib and end up with a StormLibRAD.lib file, which you add to the "linker input (additional dependencies)" (visual studio project properties). Also make sure the "additional include directories" and "additional library directories" (in mpqlib) contain the path to the header files/lib file of the stormlib project, or visual studio might not find them.


Title: Re: MpqLib
Post by: qcz on August 24, 2010, 07:09:03 PM
Hi! Thanks for the answer. The problem was VS2010, it was unable to convert the StormLib project files to it's new version, therefore no compile, and the lib files were never existed. After I installed C++ stuff for VS2008 (I used it only for C# so far), I was able to compile StormLib, and after some modifications to MpqLib (adding new compression methods, changing some calls), the new MpqLib was born :)

It works like a charm, even the compression problem is gone (So there were no problems on my side, it was probably a StormLib-related problem), and it handles SC2 files, too. I'll do some other stuff with the MpqLib code (i had to remove some SFileGetFileInfo calls temporary, and replace some as the function's signature has changed), and post here the new version if anybody interested.


Title: Re: MpqLib
Post by: Magos on August 25, 2010, 08:58:05 AM
Nice going  :)


Title: Re: MpqLib
Post by: qcz on August 27, 2010, 11:21:35 AM
Here it is, MpqLib 1.1:
http://stuff.qcz.hu/ (http://stuff.qcz.hu/)

Changelog:
- Updated to .Net 4.0
- StormLib updated to 7.02
- Added support for (attributes) file on archive creation
- Added support for new compression methods (Sparse, LZMA, ADPCM Mono, ADPCM Stereo)
- Added HashTableSize attribute to CArchive (get, set). Using this, you can change hash table size of the archive.
- CFileStream CompressedLength attribute renamed to CompressedSize (StormLib uses Size instead of Length)
- Added Size attribute to CFileStream  (get)
- ZLib and BZip2 compression problems resolved on StormLib update


Title: Re: MpqLib
Post by: LittleYoschi on September 12, 2010, 10:35:19 AM
Hi,

Currently I am working on a project where I want to read Starcraft2 replays, the listfile's for example:

replay.game.events
replay.sync.events
replay.load.info
replay.smartcam.events
replay.initData
replay.details
replay.attributes.events
replay.message.events

These are the listfiles. Unless I come Nich already, but I can not get their content read or evaluate.

The goal is simply to read out the basis of data from the replay the following:

Player name
Mapname,
Graphic map (tga is saved in the replay)
Duration of the game,
the chat during the game and so on.

These data should I be able to read from the replay, but I'm just not there ran because I do not unfortunately know how to open them or to their content rankomme precisely, each experiment has so far failed: (

I hope you can help me as possible.
For I will read this data and also for example the map graphic export.

Hopeful
LittleYoschi

P.s. I am a vb.net developer ^^


Title: Re: MpqLib
Post by: Magos on September 13, 2010, 09:50:19 AM
I don't know much (if anything :D) about SC2's fortmats. Check out qcz's version of mpqlib above, as he added SC2 support.


Title: Re: MpqLib
Post by: LittleYoschi on September 13, 2010, 10:57:20 AM
I already use version from qcz's version but I can not get to the data in the tables are in the archive.
So my problem is how do I get closer to the exact dates, because I can not simply extract because I can not read them so I know what is inside, such as, for example, the tga graphic for the map.
I hope you understand what I mean


Title: Re: MpqLib
Post by: GeMan on September 26, 2010, 09:46:35 AM
Hey thanks qcz for the SC2 update I had it working great but after the latest patch is stopped working for new replays. I have been scouring the source trying to figure out what I can do to correct it but no luck so far. Any plans on an update or anyway you can point me in the right direction? Thanks!


Title: Re: MpqLib
Post by: tomek320 on September 29, 2010, 01:01:45 AM
Magos can u explain me how to import and how work ur tool for mpq , becouse i think that i can import file in ur program and automatic on w3 mpq ??but if i erase file i cant get it back?
i mean that work for that map only or it is pernamently???


Title: Re: MpqLib
Post by: Magos on September 29, 2010, 09:16:53 AM
You have to decide yourself if you want to import/remove files in your map (which is an mpq) or in the main wc3 mpq (not recommended).


Title: Re: MpqLib
Post by: tomek320 on September 29, 2010, 11:09:40 AM
I want only to map but i dont know how ..
and thx for fast replay


Title: Re: MpqLib
Post by: GeMan on September 29, 2010, 11:21:52 AM
Man we are at the point of paying someone to get us a working dll that will parse SC2 replays for the current version of SC2. It's just beyond our scope at this point. If anyone is willing to do some work for us, Magos or anyone else. Or if you can point me to qcz please send me a message.

Thanks!
-Greg


Title: Re: MpqLib
Post by: tomek320 on September 30, 2010, 01:56:41 AM
i download ur list or what is that is but i dont know where to extract , and i extract to one file and it dont work i dont know with what to open


Title: Re: MpqLib
Post by: Magos on September 30, 2010, 09:31:10 AM
This is not a program (executable) that you can run. It's a library used when programming in dot net, like C#.


Title: C#
Post by: vercas on October 02, 2010, 05:43:52 AM
Is it okay if I attempt to convert the source code to C#? I am in serious need of fixing a retarded bug (that I cannot trace) in the constructor of CArchive too, and I must also aid in making this library by-the-standards, because I am using it.


Title: Re: MpqLib
Post by: Magos on October 03, 2010, 03:54:40 AM
That's ok, if you keep the GPL intact.


Title: Re: MpqLib
Post by: vercas on October 14, 2010, 12:13:12 PM
Cool... But now that I look at it... It's kinda hard because I am not really used to C++ header files stuff... >_<

BTW, If I attempt to convert it to .NET 4, I get 6 errors C1010.
Quote
Error    2    error C1010: unexpected end of file while looking for precompiled header. Did you forget to add '#include "StdAfx.h"' to your source?    C:\Users\Alexandru\Documents\Visual Studio 2010\Projects\MpqLib\MpqLib\Mpq\Archive.cpp    525    1    MpqLib

Any idea? O___O
________________________

Anyway... After successfully (i think) tracking down the error, I've come up to a point where "StormLib.h" cannot be found.
________________________

Or can you just re-compile the DLL for .NET 4? :D


Title: Re: MpqLib
Post by: Magos on October 15, 2010, 10:16:04 AM
You need to add a reference path to where stormlib.h is (visual studio can't find it).


Title: Re: MpqLib
Post by: vercas on October 25, 2010, 12:50:37 PM
Oh, cool! :D

Edit: Same errors. :( (Lost the source where I fixed the errors... And I forgot how to fix 'em.)

Edit2: Seriously, please recompile the project for .NET 4.0. :(

Edit3: http://www.zezula.net/en/mpq/stormlib/sfilesethashtablesize.html
Now you can change hashtable size.
I think you should add a feature to double the size when the hashtable is full. (Just like a List!)

Edit4: I beg you!


Title: Re: MpqLib
Post by: vercas on October 31, 2010, 03:56:08 AM
A nice feature would be write-able CFileStream's! :D

Edit: Cannot rename "file" to "File"?


Title: Re: MpqLib
Post by: Magos on October 31, 2010, 04:37:05 PM
I sadly do not have the time for further development on this project currently.
The source is available though if anyone is interested in developing it.


Title: Re: MpqLib
Post by: vercas on November 03, 2010, 08:18:55 AM
If it was in C#... :(


Title: Re: MpqLib
Post by: vercas on November 11, 2010, 09:14:03 AM
I beg you to recompile it for 4.0! I really to be able to target 4.0, because another important (not open source) library I need is for 4.0 only!
Moreover, it's the latest version!


Title: Re: MpqLib
Post by: vercas on November 26, 2010, 12:58:56 PM
Please!


Title: Re: MpqLib
Post by: Magos on November 27, 2010, 03:33:27 AM
Ok, I made an attempt (pretty untested).
Make sure you put the target of your project to "x86" instead of "any cpu", or it won't work on 64-bit systems.


Title: Re: MpqLib
Post by: vercas on December 11, 2010, 09:38:03 AM
Ok, thanks!
I will test this right after I get my computer back from my uncle... My BIOS got virused, but, happily, I have managed to back up my projects on another HDD right before that! :D


Title: Re: MpqLib
Post by: vercas on December 17, 2010, 11:25:40 AM
So far so good!