[REL] Riiak's DotP 2014 Deck Builder v1.5.0.3

[NeoAnderson]

As always you're right if you want my opinion should be as the follow :
1. About the duplicates it coulb be implemented as multioption search. So the user can decide what he should hide.
Option flag 1 : filename
Option flag 2 : cardname
Option flag 3 : multiverse id
Option flag 4 : code matching

2. About the problem to decide which one to show, i should manage it with a priority list where the user sort the wad installed. So the first will be showed the others will be hided.
2b. About the differences we know that each card could be implemented by different ways, for this reason the priority list will be helpful, because anyone of us could have different favorite mods, so each one would search for his own favorite first, if not avaible for the others.
2c. Could be useful to change the text colour of the raw when there are duplicates hided.

3. About the wad compactor, was just to say that sometimes would be better to have a Light game loading. There are a lot of useless data loaded by dotp because of this issue.
An idea would be to agree in a common cards developing removing from personal mods already programmed and full working cards, obviously for many reasons will be hard to do.

Anyway i really appreciate your work, your deck builder is very usefull.
If i have to find an aspect to improve i surely will look about the memory usage. I found that if you update the session the deck builder will drain slowly a lot of memory. Also during other circumstances this happens, sometimes the system asks to Close the program because of a memory shortage.

Hope you will find useful some suggestions, my friend keep in touch, and keep you amazing work go forward.

Have a nice day.

[RiiakShiNal]

As always you're right if you want my opinion should be as the follow :
1. About the duplicates it coulb be implemented as multioption search. So the user can decide what he should hide.
Option flag 1 : filename
Option flag 2 : cardname
Option flag 3 : multiverse id
Option flag 4 : code matching

I see little point in trying to mark duplicates based on Multiverse Id due to many modders prefixing the actual Multiverse Id and other modders using either made up Ids or those from magiccards.info or some other online database instead of from Gatherer. So the intended functioning of hiding duplicates based on Multiverse Id would not function properly (it would be worse than filtering based on FILENAME). For example say one modder codes a Search for Tomorrow and gives it the Multiverse Id 108802 (Gatherer) then a different modder who uses the prefix 108 codes a card and gives it Multiverse Id 108802 (802nd card that modder made if starting from 001) if filtering on Multiverse Id the Search for Tomorrow may fail to appear because it was superseded by the other card based purely on Multiverse Id. This problem likely will not occur with CARDNAME or FILENAME because the cards would have different names and that is part of both the CARDNAME and FILENAME.

2. About the problem to decide which one to show, i should manage it with a priority list where the user sort the wad installed. So the first will be showed the others will be hided.
2b. About the differences we know that each card could be implemented by different ways, for this reason the priority list will be helpful, because anyone of us could have different favorite mods, so each one would search for his own favorite first, if not avaible for the others.
2c. Could be useful to change the text colour of the raw when there are duplicates hided.

About this part, I'm going to have to say no. A user created priority list would just add too much complexity at this point. Coming up with a set of fixed rules would be about the limit at this point in time and even that will take quite a bit (the Deck Builder was not coded for filtering on other items in the list and any such filtering would mean a performance and memory hit which could be substantial).

3. About the wad compactor, was just to say that sometimes would be better to have a Light game loading. There are a lot of useless data loaded by dotp because of this issue.
An idea would be to agree in a common cards developing removing from personal mods already programmed and full working cards, obviously for many reasons will be hard to do.

Creating a common core has come up for every DotP game and for virtually every DotP game it has failed. About the only way to create a common core and distribute it properly is for ALL modders to work together on a single code base in a single repository and have 1 or 2 modders that are the QA/Release people that check all the changed cards and determine when a new version/release is made.

This does have some pros in that users will be able to expect consist quality in releases and that everything has been checked at least once. However, there are many more cons including making it difficult for new modders to join in and contribute (not to mention trying to explain to them how to use the repositories), it also puts a lot of burden on the QA/Release people as there could be hundreds of cards to check between versions. Also the larger a core becomes the more each user has to download with each new version. You start putting everyone's mods together and it could get up to several hundred megabytes or even up to a gigabyte or more. There is also the required communication between all of the modders working on the project which can be time consuming in and of itself.

While a common core is a sound idea, it is not necessarily practical. We still have people requesting that the Deck Builder build stand-alone wads that contain all required cards, functions, images, etc... which would only further the problem with duplicates. There are released mods here on the forums that are not compatible with other mods due to out-dated copies of cards and functions present in one or the other.

Anyway i really appreciate your work, your deck builder is very usefull.
If i have to find an aspect to improve i surely will look about the memory usage. I found that if you update the session the deck builder will drain slowly a lot of memory. Also during other circumstances this happens, sometimes the system asks to Close the program because of a memory shortage.

If this is happening with the 64-bit version then it is likely you don't have a lot system memory in which case you can turn off the image caching by putting this line in the Settings.xml:

Code: Select all

<MaintainImageCache>False</MaintainImageCache>

By default the 64-bit version will use an image cache to improve performance while the 32-bit version will not use the image cache to keep memory usage to a minimum. Though the image cache can be turned on for the 32-bit version by putting this line in the Settings.xml:

Code: Select all

<MaintainImageCache>True</MaintainImageCache>

As for other optimizations those are relatively unlikely due to how C# manages its memory (I don't have deterministic control over objects). Though in the same respect C# ensures that there are no memory leaks in the program and allows for faster development (and more features sooner) than would have been possible with C++.

[NeoAnderson]

Is always nice to interact with you, i like your detailed explainations,
you have made the utility and you know what can be done to improve it.
About the priority list i didn't tought it could be so complex to implement, i see it just as the "files order" to use when you read the wads.
The program already have the possibility to sort the data by wad column, so this kind of information is already used.
Anyway as i say, you know what is the better way.
About the memory shortage yeah my pc need a boost, the system is a 2 gigabyte ram based, 64 bit o.s. Windows 7. The problem is many times i i have too many programs opened at same time, heavy programs, as like photoshop.

[RiiakShiNal]

Is always nice to interact with you, i like your detailed explainations,

I have found it is usually pointless to give simple answers that do not give all necessary information in an easy-to-understand way because then I get asked the same questions over and over again by the same people because they don't understand the reasoning behind such answers.

On the flip side it also allows for people to look at my reasoning and if they see a problem or mistake in my reasoning they can either make their own informed decisions or point out the mistake in my reasoning and I can learn from it as well.

you have made the utility and you know what can be done to improve it.
About the priority list i didn't tought it could be so complex to implement, i see it just as the "files order" to use when you read the wads.
The program already have the possibility to sort the data by wad column, so this kind of information is already used.
Anyway as i say, you know what is the better way.

The Wad column is not really used that way. First I put all cards into a single list, then that list may be filtered through a predicate function (which does not rely on anything else in the list) to create a new list which is then given to the DataGridView for displaying the cards. After that the sorting is handled by the list implementation either by simple text or numeric comparison (which can be done fairly quickly, though slower on text compares).

As such there currently is no "files order" in the Deck Builder. To do a priority listing for hiding duplicates first a new list must be constructed (Memory hit) using the original list and either:

• Sorting the list on the duplicate comparison field so that consecutive objects with the same value for the duplicate comparison field are "duplicates". (Performance Hit)
• Use an additional set collection to store the values from the duplicate comparison field to check for duplicates as this round of filtering is done. (Memory Hit)

And this process can be done either before or after predicate filtering function which handles basic/advanced filtering. Then if I have a user set priority then I have to keep track of yet another list or value set which has to be compared for every sort/check when trying to filter duplicates which further increases complexity and introduces yet another performance hit and possibly memory hit.

A fixed priority rule set can be hard-coded and optimized (to have as small a performance and memory hit as possible) while a user-set priority list can't be optimized at all because it can change at any time (not to mention it will change every time a new wad is added or removed even if just to automatically remove from the list or add to the end of the list). Additionally a user-set priority list would mean more for the user to understand and set in the Deck Builder for it to run as expected, while we still have lots of users who don't even understand that they should change the Id Block of the Builder on first start-up to prevent deck id collisions. All you have to do to see this is look around at how many people are still using Id Block 1000 (and sometimes releasing mods using that block).

About the memory shortage yeah my pc need a boost, the system is a 2 gigabyte ram based, 64 bit o.s. Windows 7. The problem is many times i i have too many programs opened at same time, heavy programs, as like photoshop.

Because you only have 2 GB of memory you are basically running under the same restrictions as the 32-bit version of the program runs under so you should either turn off the Image Cache (as mentioned in my previous post) or simply switch to the 32-bit version.

There is almost no advantage to running a 64-bit OS on a computer with less than 4 GB of RAM. This is because 32-bit Operating Systems allow for up to 4 GB of addressable RAM (this includes both System RAM and Graphics RAM), while 64-bit Operating Systems allow for up to 16 EB (16 Exabytes = 16,777,216 TB) of addressable memory. However, 64-bit addressing means that all addresses are 2x the size (so 2x the memory) of addresses in 32-bit Operating Systems. The more pointers/references the more the difference in memory usage between a 32-bit program and a 64-bit program.

[NeoAnderson]

Perferctly understand everything

About the complexity of your tool, i know, not always more features make it user friendly for basic users

About my PC you're right my friend, unfortunately i am not working in this period don't let me think about this...my life is just upside down right now.
Hope to come back to a normal life soon, and then, I will spend again some money for a needed upgrade.

[HELLO2468]

Hello to all.

I am receiving an unhandled exception warning (Most likely due to my recent inclusion of GrovyleXShinyCelebi Duels 2012/13 imported decks), but is there anyway to ignore this error or will the deck builder only work with 2014 decks. (Note: The deck builder previously worked flawlessly whilst using Fireminds decks)

The cards still show up and I can still load decks etc but every time i perform this or any action this unhandled exception shows.

Below is (or was) the error gibberish that may tell you something, but is totally incomprehensible to a simpleton such as myself!

Apparently slightlymagic is telling me that this post looked - and I quote "too spamy for a new user"... I have never been insulted so much in my life!!

To rectify this annoyance, I have therefore attached the error report instead.

...too spamy for a new user huh!?...

p.s Anyway good work with the builder btw, made a couple of fun decks with it.

cheers. cole.

[RiiakShiNal]

Looking at the error it seems as though you have an incomplete wad as the error stated that it tried to read past the end of the file. This probably means that your download cut off prematurely and you did not get the entire wad. You can also interpret the error as one of your wads is corrupt.

The Deck Builder will only work properly with files made for DotP 2014, but the two mods you mention Grovyle's 2012 & 2013 deck imports were made for DotP 2014 so there shouldn't be a problem there unless your download was cut-off or corrupted (or in the unlikely case that the wad was not properly created).

As for the "too spamy for a new user" that message will go away once you have made 5 posts. It is a protection method to try and prevent spammers from quickly creating accounts then filling the forums with spam (though it can be annoying for new legitimate users).

[HELLO2468]

Hey thx for your quick reply.

Yes that sounds most likely that the download may have cut of early, I did have a few issues with that evil daemon they call the 'Internet' y/day!

So I will have to try and locate what deck/wad that is suspect, but at least i know what to look for now thank you.

Only joking about 'spam' message though, I understand its reasoning!

Thanks Again.

Cheers. Cole

[wtb300]

Im having to clear all filters and search for cards manually in the builder. searching by name, sub types and abilities yields no results yet with a manual search the card im looking for is always there. Heres the error.txt i get on startup.

11/27/2013 4:09:18 PM: Log Opened.
11/27/2013 4:09:18 PM: Low: System.Xml - 'GetEffectX' is an unexpected token. The expected token is '"' or '''. Line 60, position 88.:
at System.Xml.XmlTextReaderImpl.Throw(String res, String[] args)
at System.Xml.XmlTextReaderImpl.ParseAttributes()
at System.Xml.XmlTextReaderImpl.ParseElement()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at RSN.DotP.CardInfo.ParseXML(String strXML)
at RSN.DotP.CardInfo..ctor(String strFilename, String strXML, String strWad, GameDirectory gdData)
at RSN.DotP.WadWrapper.LoadCards(FileStream fsInput, GameDirectory gdData)
Extra Information:
Unable to load card: POLUKRANOS_WORLD_EATER_973001.xml in DATA_CORE_972

11/27/2013 4:09:18 PM: Low: Card (THALLID_9721370352 in DATA_CORE_972) has a multiverse id that is problematic: 9721370352

11/27/2013 4:09:18 PM: Low: Card (THALLID_9722370352 in DATA_CORE_972) has a multiverse id that is problematic: 9722370352

11/27/2013 4:09:18 PM: Low: Card (THALLID_9723370352 in DATA_CORE_972) has a multiverse id that is problematic: 9723370352

11/27/2013 4:09:18 PM: Low: Card (THALLID_9724370352 in DATA_CORE_972) has a multiverse id that is problematic: 9724370352

11/27/2013 4:09:23 PM: Low: Card (JACE_BAD_ASS_111000222333 in DATA_DLC_DECK_BUILDER_CUSTOM) has a multiverse id that is problematic: 111000222333

11/27/2013 4:09:52 PM: Log Closed.


I have removed the DATA_CORE_972 and my jace bad ass as he is unused now and all decks that required DATA_CORE_972. i now get no error.txt but the problem still persists.

[RiiakShiNal]

That error log has nothing to do with your problem of not being able to search, but I'm not sure why you aren't able to search. The error log reports one card that is not able to be loaded (POLUKRANOS_WORLD_EATER_973001.XML in DATA_CORE_972.WAD) due to malformed XML and then 5 cards that have invalid Multiverse Ids (they are outside the 32-bit integer space for ids), but again this has nothing to do with searching.

I assume you are using the current version (v1.3.2.0), but are you trying to search with the basic or advanced filters? The more details about how exactly you are trying to search would also be appreciated such as in which language are you filtering in (the Deck Builder uses the language selected in the Options screen for text filtering)? Are there specific examples which fail to produce the correct behavior? Were there special characters in the search string or in the card that you were searching for?

[wtb300]

It was basic filters in english and all filter options ie Abilities I searched for defender and had no results. By name I searched for Squadron Hawk with no results so I closed the builder and I deleted the Settings.Xml then reopened the builder and set the directory, and reset my UID back to 300 and everything is working fine again. If I had thought to save the settings.xml I would post it but if it happens again I will include it in my post.

[RiiakShiNal]

Well, it's good that it's working now, but that is strange, in the meantime I will look over the basic filters again and see if I can find the problem.

[Springwight]

Hey there. I really love your program. I encountered a small problem where if you have many rows in the cards DataGridView, especially if they contain images for mana, scrolling gets really sluggish and lags a lot. This is a problem that's inherent to WinForms and can't really be fixed completely.

However, I've managed to improve the performance significantly with several rather small fixes. I'd upload the fixes myself, but since there is no source control, things would just get confusing. So I'll explain them instead.

• There is a protected boolean property in the DataGridView called DoubleBuffered. Setting it to true improves the performance on some machines. You can do this by using reflection (this is actually not an uncommon practice and doesn't break anything AFAIK). Here is the code:
  

  Code: Select all

  dgvCards.GetType().GetProperty
                ("DoubleBuffered", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(dgvCards, true, null);

• Implementing virtual mode on the DataGridView. Basically, you circumvent ordinary data binding and implement a custom CellValueNeeded handler which supplies the cell values to the control when it asks for them, based on a separate cache managed by you. There are some MSDN references on the topic. Here is my implementation. Note that you need to change a few other things in the code for it to work.
  

  Code: Select all

          private void dgvCards_CellValueNeeded(object sender, DataGridViewCellValueEventArgs e)
        {
           
            if (e.RowIndex == dgvCards.RowCount - 1) return;
            var list = (SortableBindingList<CardInfo>)m_bsCards.DataSource;
            var current = list[e.RowIndex];
            switch (e.ColumnIndex)
            {
                case 0:
                    e.Value = current.LocalizedName;
                    break;
                case 1:
                    e.Value = current.Filename;
                    break;
                case 2:
                    e.Value = current.ColourText;
                    break;
                case 3:
                    e.Value = current.CastingCostImage;
                    break;
                case 4:
                    e.Value = current.CastingCost;
                    break;
                case 5:
                    e.Value = current.ConvertedManaCost;
                    break;
                case 6:
                    e.Value = current.LocalizedTypeLine;
                    break;
                case 7:
                    e.Value = current.Power;
                    break;
                case 8:
                    e.Value = current.Toughness;
                    break;
                case 9:
                    e.Value = current.Token;
                    break;
                case 10:
                    e.Value = current.Rarity;
                    break;
                case 11:
                    e.Value = current.Artist;
                    break;
                case 12:
                    e.Value = current.Expansion;
                    break;
                case 13:
                    e.Value = current.PresentInWad;
                    break;

            }
        }

I'd post some links on the subjects, but it seems new users aren't allowed to do this.

By the way, if you host the source code on GitHub or similar, I could implement this modification, and also try to add some persistent caching capability for the images. Profiling indicates that squish does 96% of work when initializing the application.

[RiiakShiNal]

Hey there. I really love your program. I encountered a small problem where if you have many rows in the cards DataGridView, especially if they contain images for mana, scrolling gets really sluggish and lags a lot. This is a problem that's inherent to WinForms and can't really be fixed completely.

Yeah, I do know about the sluggish performance on the DataGridView and know it is partially related to WinForms, I've been working on other areas of the builder as of late and hadn't gotten around to improving the performance there.

However, I've managed to improve the performance significantly with several rather small fixes. I'd upload the fixes myself, but since there is no source control, things would just get confusing. So I'll explain them instead.

I'm paying close attention now as performance improvements are always appreciated especially when they are significant improvements. As for version control, I've been managing mine locally rather than using public resources. I actually had not quite envisioned how large this project of mine was going to become.

• There is a protected boolean property in the DataGridView called DoubleBuffered. Setting it to true improves the performance on some machines. You can do this by using reflection (this is actually not an uncommon practice and doesn't break anything AFAIK). Here is the code:
  

  Code: Select all

  dgvCards.GetType().GetProperty
                ("DoubleBuffered", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(dgvCards, true, null);

I know about (and have used in the past) Double Buffering, but I did not know that it would provide much of a performance increase on some machines in .Net 4. It is mainly used to reduce flickering when redrawing a surface (image, control, etc...) and works by having the surface drawn to system memory (rather than graphic memory) and then transfers all at once when drawing is finished (I actually use this technique for the Build Image functionality). The downside is that it requires more memory, though for the DataGridView that is probably negligible.

I'll run some tests and as long as the increase in memory is negligible (as I would suspect) then I'll put it in to the next version (as long as it doesn't actually cause a slow down, which I don't expect it will). And no it shouldn't break anything, I use reflection elsewhere without problem so it shouldn't be a big deal here.

• Implementing virtual mode on the DataGridView. Basically, you circumvent ordinary data binding and implement a custom CellValueNeeded handler which supplies the cell values to the control when it asks for them, based on a separate cache managed by you. There are some MSDN references on the topic. Here is my implementation. Note that you need to change a few other things in the code for it to work.
  

  Code: Select all

          private void dgvCards_CellValueNeeded(object sender, DataGridViewCellValueEventArgs e)
        {
           
            if (e.RowIndex == dgvCards.RowCount - 1) return;
            var list = (SortableBindingList<CardInfo>)m_bsCards.DataSource;
            var current = list[e.RowIndex];
            switch (e.ColumnIndex)
            {
                case 0:
                    e.Value = current.LocalizedName;
                    break;
                case 1:
                    e.Value = current.Filename;
                    break;
                case 2:
                    e.Value = current.ColourText;
                    break;
                case 3:
                    e.Value = current.CastingCostImage;
                    break;
                case 4:
                    e.Value = current.CastingCost;
                    break;
                case 5:
                    e.Value = current.ConvertedManaCost;
                    break;
                case 6:
                    e.Value = current.LocalizedTypeLine;
                    break;
                case 7:
                    e.Value = current.Power;
                    break;
                case 8:
                    e.Value = current.Toughness;
                    break;
                case 9:
                    e.Value = current.Token;
                    break;
                case 10:
                    e.Value = current.Rarity;
                    break;
                case 11:
                    e.Value = current.Artist;
                    break;
                case 12:
                    e.Value = current.Expansion;
                    break;
                case 13:
                    e.Value = current.PresentInWad;
                    break;

            }
        }

Ahh virtual mode, this would be the next logical step now that I handle all the sorting and if implemented right should be a significant performance increase as you stated.

The code you provided here though won't work right at all. If you look carefully at how I've implemented the columns (including saving and loading) for the DataGridView you will notice that you have only covered the base case as column definitions can be overridden using the Settings.xml including the addition of new columns so a more generic approach would need to be taken to account for this. Also because I do not allow new rows in the DataGridView, your "if" at the beginning of the procedure would highly likely prevent the last item in the list from being shown.

I'd post some links on the subjects, but it seems new users aren't allowed to do this.

The new user restrictions will go away once you've made 5 posts, until then you can still post links by altering them so that they don't appear like links. For example you can omit the "http://" and any "www" instances at the beginning as most browsers will add those automatically you can replace slashes with some unused character in the link such as ">" and replace the periods with some other unlikely character such as "_" or ",". So for example a link to a MSDN KB article on implementing virtual mode code change from:
http://msdn.microsoft.com/en-us/library/15a31akc%28v=vs.90%29.aspx
to:
msdn,microsoft,com>en-us>library>15a31akc%28v=vs,90%29,aspx

It's a little more work for both poster and reader, but it will allow you to get past the new poster restrictions to post links.

By the way, if you host the source code on GitHub or similar, I could implement this modification, and also try to add some persistent caching capability for the images. Profiling indicates that squish does 96% of work when initializing the application.

I may end up using a public source code control site later on, though right now I'm trying to get some new features (mostly requests) in so I can get a new version out.

As for the persistent image caching, I already have some in the Builder though for due to memory restrictions the 32-bit version has this disabled by default. Even still casting cost images are cached (for example all cards with a casting cost of will all refer to the exact same image and it is only built once) regardless of which version you are using to both reduce some of the memory footprint and to speed things up a bit. Caching of the individual mana symbols, however, is only done if full image caching is enabled (hard coding those in as cached would provide a decent speed increase for when the full image cache is disabled). The 64-bit version has full image caching enabled by default to speed things up as well. The image cache can be forced on (True) or off (False) by setting (or adding) the appropriate value in the Settings.xml:

Code: Select all

<MaintainImageCache>True</MaintainImageCache>

So short answer:
I see and have noted (and appreciate) your suggestions for performance improvements to the Builder and I will review them in detail and add appropriate versions of them as I can.

[Springwight]

I know about (and have used in the past) Double Buffering...

Yup, part of the problem (as I saw it on my machine) was that it redrew the GridView very slowly. (My machine isn't really a slouch either).

The code you provided here though won't work right at all...

Ahh, yeah, I suspected I had gotten some things wrong. I didn't expect the code to be used as-is, so I didn't test it beyond to see that the columns render correctly and I can get things like sorting to work.

As for the persistent image caching...

I actually meant storing the images (and possibly other WAD file content) on the disk itself, in converted or decompressed or unpacked form, possibly in separate image or data files, or preferably using a database (I've had good experience at using free NET object databases for similar purposes). This is so you don't have to parse and convert the same images every time you start up the program, which as I've mentioned, makes up for most of the work done at start up. Most WAD files aren't going to change much for any specific user, after all.

If the images don't have unique identifiers, you could make sure the cache is up to date by using some efficient hash function (on the whole WAD file, or on individual blocks), or perhaps just storing some WAD file metadata. This is, of course, a lot more work than just tweaking the grid view. The issue is that startup can take a long time if you, e.g., load one or more of the complete sets and other mods. In particular, it takes ages if you run it with debugging enabled.

I see and have noted (and appreciate) your suggestions for performance improvements to the Builder and I will review them in detail and add appropriate versions of them as I can.

Sure, I'm happy to help This is really a great application. I don't know if I could play the game without it.

By the way, the spam detector you have here is really something. I can't even type (dot)NET.