DotP 2014: Potential Community Wad

[Xander9009]

Hey, thanks for doing this!

Sorry in advance for my newbness. I just found out you could add decks to DotP 2014 this week, so I'm a bit behind. I'm looking to add two cards, Expedition Map and Wing Splicer. I was hoping that the art for those cards would be in this big community art WAD - but I'm not sure how to even open the art wad to look around.

I realize I may be derailing this thread, so if there's a better place to find this information I'd appreciate being pointed in the right direction. I just figured that the art for those cards was somewhere in this community pack.

You can open a wad file using Gibbed Tools. You can find RiiakShiNal's modified version here.

Yes, they're pretty much definitely in there. You don't need to open it to look around. Go to http://gatherer.wizards.com/Pages/Default.aspx and search for the card. In the URL, note the MultiverseID. Add "CW" to the beginning of that, and use that for the art id in your card's xml. CW193405 should make the right picture show up for Expedition Map and CW233037 should work for Wing Splicer.

EDIT: Just checked, they're both there under those IDs like they should be.

[darist]

Thanks! This is awesome.

[AngelLestat]

you are right, if everybody start to upload their decks, it will be a mess. I have already 38 :S

But your idea seems great. I hope that in the future we would be able to have that single wad without duplicate cards.

[eldrazi]

This seems like a job for bittorrent sync. I'm a new user so I can't hotlink, but it's easily installable at ninite

[RiiakShiNal]

This seems like a job for bittorrent sync. I'm a new user so I can't hotlink, but it's easily installable at ninite

You could theoretically use Bittorrent Sync for distributing the WADs, but that may be less efficient than using a simple script for checking and downloading new versions. It would definitely not be a good solution for the base files used for development as there would be no code control, versioning, or security (which would allow for easy corruption of the repository). For development purposes the repository would need strong code control and versioning to ensure that the repository isn't permanently poisoned. Security could be used to make sure only actual contributors have commit access which also reduces the chance of poisoning, but also restricts who will use it some.

Bittorrent Sync would simply distribute the bandwidth usage among all accessing it so that everyone who gets the files could also be responsible for uploading to others as well as keeping things sync'd. However, then you have other problems which I have mentioned before such as if the Sync is in the process of getting the new file and you run the game what happens? I see 2 distinct possibilities, 1 The game runs with the old file (which may or may not be buggy) and is unable to connect to other people online due to running a different version of the WAD or 2 the game tries to read in the as yet incomplete file (or is completely denied access) and leads to either crashes or not using the WAD altogether in which case again can't connect to people online (or if custom decks are installed using cards from the WAD will cause the game to crash because it can't find the cards).

There is currently no readily available easy solution. Even if there were such a solution it would still require getting all the contributors to use it which is a separate difficult issue.

[Xander9009]

I'm not sure why I wasn't informed of these replies. Either way, I've never heard of bittorrent sync, but seeing as Riiak has and has already pointed out why it wouldn't be a great option, I'm inclined to go with his opinion.

The way it's working now is already fully working and incredibly simple. Nothing is left to do to have it up and running except to get people to actually use it (this doesn't mean we won't discover things that need changed, though). But since no one has really shown interest in it, I'm not particularly hopeful for 2014.

It might change when 2015 is released. Perhaps then, I'll start a discussion in the main forum where the majority of people are more likely to see it discussing whether or not to use it. As it stands, getting others to adopt it, as Riiak said, is the biggest issue, and I'm just hoping they'll be more amenable to trying it if it's with a new release of the game.

Riiak, I may have a solution to that issue. AHK has a simple method of checking when a process exists. As it's working right now (if I remember correctly), it simply waits until the files are all ready and then moves them quickly to try and give the shortest window possible where things might mess up. I could have it check if the game is running or starts up, and if it's running or starting, it could play a sound to let them know the game might not work correctly until they restart it. In the time it would take to shut down and restart the game, the transfer should complete (the movement of the files should only take a couple of seconds). I considered a message box, but that would minimize the game. The sound is noticeable without being intrusive. Thoughts?

[RiiakShiNal]

Riiak, I may have a solution to that issue. AHK has a simple method of checking when a process exists. As it's working right now (if I remember correctly), it simply waits until the files are all ready and then moves them quickly to try and give the shortest window possible where things might mess up. I could have it check if the game is running or starts up, and if it's running or starting, it could play a sound to let them know the game might not work correctly until they restart it. In the time it would take to shut down and restart the game, the transfer should complete (the movement of the files should only take a couple of seconds). I considered a message box, but that would minimize the game. The sound is noticeable without being intrusive. Thoughts?

That could definitely help, but it is still not a complete solution (the sound can be ignored, possibly mistaken for something else, not heard if muted or the user is hard of hearing/deaf, etc...). The best solution would be that when it finds a new version that if the game is not currently running it should prevent the game from being run (with a message box stating that it is the process of being updated, possibly with estimated time remaining) until the update has downloaded (though I don't know if that is possible with AHK), if the game is already running it can download the update (in the background) to a separate directory and move it in when the user exits (this part should be possible given what you have said about AHK being able to detect a process).

[Xander9009]

Alright, I did some testing, and I find it extremely unlikely that will ever be an issue. How it works, Google Drive downloads the file to the computer. The program will check one once every N seconds (you set it to whatever you want) to see if there's a new version on your computer. If it finds one, it copies the new version to the game directory with the .bak extension. The first time it checks for the process and finds it's not running (it'll check every 2.5 seconds), it will delete the old version and rename the new one. With a 3 GB file, it takes 30 seconds to copy, and 2 seconds to delete/rename. With everything (including the game's core files and all expansions, plus almost every mod I've downloaded) excluding the art folder, it only came up to 140 MB (75 MB without core files), and that transfers in about 5 seconds. So, the progress bar would be unnecessary. (For clarity, I tested with a 3 GB file just as an outer limit, should the process one-day include event the art on a regular basis.)

The fact that the time is so short between knowing there's a transfer needed and the transfer being completed means preventing the game from running is also probably an unnecessary step. If you happen to decide to run the game within that 5 second window, then you'll be running on the old version which will be missing the cards made/fixed within the last day, but that's unlikely to cause any major issues. With that in mind, I really don't like preventing the game from running; however, if you think I should make sure it's noticed, I can go ahead and put in a message box that should minimize the game and let them know they should exit, wait for confirmation, and then play the game. This way, it doesn't affect their actual ability to play, but it'll make sure they know about it.

EDIT: I also just thought of the fact that the art file will probably update a little more often than I was originally thinking. It'll be a rather large file, though, and many people probably wouldn't want it taking up space in the Google Drive or having 2 copies (which is how the core works). So, I was thinking I could have them keep just the version file, and when the version files don't match but there's no art wad, tell them they need to update it manually.

[MC Brodie]

I'm interested in using and/or helping out with a community wad for 2015 (if it's moddable). I just don't have any free time right now and my schedule likely won't clear up until after July.

As far as the community wad goes, would it be better to use a loose wad directory instead of a packed wad? At least with the art file it could save people from redownloading images they already have. Honestly I think it would be much better if everyone switched to the loose wad format since it's easier to add patches and updates. That's just my personal preference. Though this approach may be more prone for compatibility problems when trying to play multiplayer. If that's the case it would be nice if there were 2 downloaded versions: a packed wad and a loose directory.

Also, I think we need to be careful with what cards are included in the community wad and/or how they are implemented. Due to our limitations, a lot of cards are approximated. For the most part, the approximations are card specific and do not have a large impact on other cards. The one big example is the mana pool implementation. Some people really like the ability to play with a mana pool and some don't want to use it. I don't think the community wad should include major game changing approximations upfront. They could be compiled in an additional wad or we could try something like I did in my latest mod. I essentially had a check to see if the manual mana functions were installed and then the card interacted accordingly.


Lastly, if other modders don't want to use the community wad, I think it would benefit the community the most of we all use a single art wad. The art files take up the most space and if you have multiple mods downloaded it is the same file multiple times. I think it would be best if everyone would at least agree on a standardized naming convention and use a community art wad.

[Xander9009]

The Community Art Wad is done up to Born of the Gods. I just downloaded the latest version yesterday which should get me any newly added card images since I last messed with it, and I have scripts which automate the entire process from start to finish for organizing, renaming, and converting the images. The Art Wad that is available is already good to go, and they're all named the same way: CW[MultiverseID as found on Gatherer].

The loose files vs wad question: A loose wad would be easier to patch if you were patching manually. However, with the setup that I currently have in place (which is already 100% working, though that doesn't mean it won't need tweaks) consists of this
Modders: Drop card files or whatever other non-art files into loose-file directory where they would normally go.
Users: Do nothing.

Without any assistance from anyone, the modders' files are automatically packed into a wad, then that wad is synced to everyone who is set up correctly so they are always up to date. The only time issues are likely to arise is if more than one person tries to simultaneously modify a single file. That might cause a conflict, but it's also unlikely to happen often, and Google Drive should let us handle those conflicts. So, it shouldn't really be an issue as far as that goes.

I have considered the art problem. The main art file (the 1-2 GB worth of pictures) is very large, and syncing it to everyone every day would be a pretty harsh solution. I think you're right about using a directory approach here. In fact, after some thought, I think I'm going to go back on what I said.

I may have just figured out a solution to the whole thing. The files (including art, cards, and all other files) are on a public folder. The utility I already have that syncs the files gets modified so it instead checks against a list of files. Updated files are downloaded directly from the public Google Drive folder to the game folder. It would mean art would be as easy to distribute as any other file, and unlike the current system, you don't need two copies of any file (currently, the core wad exists in both the on-system Google Drive folder and in the game folder). It would actually be a bit slower, so I'll probably make both available and people can decide which they prefer. Since it will all be automated and actually rather simple, it should make everything work together and allow the art to be updated without downloading 1.2 GB of stuff. I'll see how AHK handles this approach and post back in awhile (it might not be tonight).

[RiiakShiNal]

For multiplayer packed WADs are better than the loose directory method. The reason for this is that different systems may recognize the files in the loose directory differently than on another system (automatically generated system thumbnails, or other automatic files generated on a system) so DotP 2014 would recognize the directory as a non-matching WAD between those two systems. However, with a packed WAD you don't have to worry about automatically generated files since the entire directory structure is a single file (this also makes version checking faster and easier for DotP since it doesn't need as many handles and open files).

For development though the loose directory method is pretty much optimal, it just causes issues with online multiplayer.

[Xander9009]

I didn't consider that (and this is why I wanted others' input lol). I actually noticed that I accidentally left in the desktop.ini file when I packed the wads just today and haven't even gotten around to making my wad packer delete those before running, yet even with that actively on my mind I didn't consider that would cause problems for multiplayer. With that in mind, I'll go ahead with what I was planning and make both available. If you're planning to do multiplayer, you'll need the wad, otherwise you could pick either one. Having both available won't change the complexity by much, so I'll go ahead and work toward getting that working.

[Xander9009]

Hey Riiak, I have a couple of questions you might be able to answer having worked on your deck builder. First, how do I go about checking to see if a folder is properly set up to act as a wad? I'm under the impression that it needs to

1. Have a properly formatted name: "DATA_DECKS_..." or "DATA_DLC_..."
2. Have a header in the folder titled @header.xml
3. @header needs to very specifically formatted with only the Source tag changing, and that tag should have the path of the folder which the @header is next to.
4. That folder needs a header titled HEADER.xml which is otherwise identical to @header.xml

Is this all? Or is there more to it? Basically, I'm writing a small utility to sync game folders between my computer and my friend's because she just got the game. I wanted to avoid the hassle of making sure we're running the same wads, and while all of the .wads are working, the unpacked, loose file directories aren't because I don't know for sure if there's more I should be checking. It'll also form the basis for the loose file syncer (which I now have to rewrite because I lost everything in my game folder ).

[RiiakShiNal]

Hey Riiak, I have a couple of questions you might be able to answer having worked on your deck builder. First, how do I go about checking to see if a folder is properly set up to act as a wad? I'm under the impression that it needs to

1. Have a properly formatted name: "DATA_DECKS_..." or "DATA_DLC_..."
2. Have a header in the folder titled @header.xml
3. @header needs to very specifically formatted with only the Source tag changing, and that tag should have the path of the folder which the @header is next to.
4. That folder needs a header titled HEADER.xml which is otherwise identical to @header.xml

Is this all? Or is there more to it? Basically, I'm writing a small utility to sync game folders between my computer and my friend's because she just got the game. I wanted to avoid the hassle of making sure we're running the same wads, and while all of the .wads are working, the unpacked, loose file directories aren't because I don't know for sure if there's more I should be checking. It'll also form the basis for the loose file syncer (which I now have to rewrite because I lost everything in my game folder ).

Close, the requirements for a directory WAD is very similar, but slightly different as compared to packed WADs.

As for naming you are correct the directory must start with either DATA_DLC_ or DATA_DECKS_. That folder must also have a HEADER.XML file which is a properly formatted WAD header (can range from very simple for a data WAD to more complex for a Content Pack Enabler). Directory WADs do not need an @header.xml file as it is unused by the game.

As such it is very simple to take an existing packed WAD and convert to a directory WAD. You simply unpack the WAD using Gibbed Tools, take the inner directory (not the one that ends in "_unpacked") and put that in the game directory. If there is already a HEADER.XML in the right place (should be the case with mods made by me or my builder) then you are done. If there is no HEADER.XML then you can take the @header.xml and rename it to HEADER.XML and put it in the proper place. Packed WADs do not need a HEADER.XML, but must have the header information embedded in the WAD (Gibbed Tools accepts @header.xml for this, but you can specify a different file using command line parameters).

If you are making the directory WAD from scratch then you will need to make the proper structure (you can omit empty directories) and create an appropriate HEADER.XML.

General Structure | Open

Code: Select all

+ DATA_DLC_1234_TEST_WAD
|- HEADER.XML
\+ DATA_ALL_PLATFORMS
 |- AI_PERSONALITIES
 |+ ART_ASSETS
 ||- ANIMATED_ILLUSTRATIONS
 ||- ILLUSTRATIONS
 |\+ TEXTURES
 | |- DECKS
 | \- PLANESWALKERS
 |- CARDS
 |- DECKS
 |- TEXT_PERMANENT
 \- UNLOCKS

Example Simple Header | Open

Code: Select all

<?xml version="1.0"?>
<WAD_HEADER>
   <ENTRY platform="ALL" source="DATA_DLC_1234_TEST_WAD/DATA_ALL_PLATFORMS/" alias="Content" order="3"/>
</WAD_HEADER>

Example CPE Header | Open

Code: Select all

<?xml version="1.0"?>
<WAD_HEADER>
   <ENTRY platform="ALL" source="DATA_DLC_1234_TEST_WAD/DATA_ALL_PLATFORMS/" alias="Content" order="3" />
   <CONTENTPACK UID="1234">
      <PD_SECTION>
         <APP_ID ID="213850" />
      </PD_SECTION>
      <CONTENTFLAGS>
         <AVATAR_CONTENT />
         <DECK_CONTENT />
         <GLOSSARY_CONTENT />
         <UNLOCK_CONTENT />
      </CONTENTFLAGS>
   </CONTENTPACK>
</WAD_HEADER>

If you need content from the WAD to override another WAD then you need to adjust the order attribute in the ENTRY tag in the HEADER to be higher than the one in the WAD you want to override.

[Xander9009]

Alright, thank you. I had everything set up checking all of this except it won't work for content pack enabling directories (and I've never seen one, so that might not be an issue anyway) and I need to move everything up a folder to ignore the @header and skip straight to the HEADER folder. Ironically, I've actually already switched between the two versions a few times over the last couple of days but I still managed to overlook the @header issue. Sometimes I feel like a dolt haha.