Moving away from cards.txt

[Rob Cashwalker]

Well, it's been a long time coming. Part of mtgrares' vision towards v2 was individual files. A realization that as we move to supporting more cards and features like sets, one text file can't contain all the information we want.

A relatively simple change has converted 4100 blocks of text in one file into 4100 files, one for each card.

The format has changed a bit, which will allow for further enhancements in the future. Without much effort, additional card properties can be supported, like multiple rarities for cards printed across sets or SVars won't have to be handled as keywords.

/res/cardsfolder/ shall now contain files of any name, however keeping with the convention used for the card images would be ideal.

Each file will look like this:

Code: Select all

Name:{Card Name}
ManaCost:{Mana Cost}
Types:{Card Types}
Text:{Card Text}
PT:{Power/Toughness}
K:{Keyword}
K:{More Keywords}
End


The order of elements is not critical, as they are named. Further refinement of our keywords and scripts will follow this pattern.

Example - abandoned_outpost.txt:

Code: Select all

Name:Abandoned Outpost
ManaCost:no cost
Types:Land
Text:no text
K:tap: add W
K:tap, Sacrifice CARDNAME: Add W to your mana pool.
K:tap, Sacrifice CARDNAME: Add U to your mana pool.
K:tap, Sacrifice CARDNAME: Add B to your mana pool.
K:tap, Sacrifice CARDNAME: Add R to your mana pool.
K:tap, Sacrifice CARDNAME: Add G to your mana pool.
K:Comes into play tapped.
K:SVar:Rarity:Common
K:SVar:Picture:http://www.wizards.com/global/images/magic/general/abandoned_outpost.jpg
End


The main drawback is not being able to search in a single file. Maintenance will become more of a chore.

The upside though is that changes and additional cards won't cause conflicts as often, and individual changes will be more visible in the SVN revision history.

Be prepared, this is at least a 2 MB update.

[slapshot5]

If we are undertaking something this big, would it be wise to make each file XML?

Something like:

Code: Select all

<card>
   <name>Abandoned Outpost</name>
   <cost>no cost</cost>
   <types>
      <type>Land</type>
   </types>
   <text>no text</text>
   <keywords>
      <keyword>tap: add W</keyword>
      <keyword>tap, Sacrifice CARDNAME: Add W to your mana pool.</keyword>
      <keyword>tap, Sacrifice CARDNAME: Add U to your mana pool.</keyword>
      <keyword>tap, Sacrifice CARDNAME: Add B to your mana pool.</keyword>
      <keyword>tap, Sacrifice CARDNAME: Add R to your mana pool.</keyword>
      <keyword>tap, Sacrifice CARDNAME: Add G to your mana pool.</keyword>
      <keyword>Comes into play tapped.</keyword>
   </keywords>
   <rarity>Common</rarity>
   <picture>http://www.wizards.com/global/images/magic/general/abandoned_outpost.jpg</picture>
</card>


The schema is flexible, and I think this would cut down on errors, as it is much more structured than plain text.

Something to discuss. Thoughts?

-slapshot5

edit: forgot some end tags

[Rob Cashwalker]

I don't think so. It may be more structured, but it's not going to provide any more benefit than what I came up with. Plus, I'm having a hard enough time uploading the cards to the SVN, at just under roughly 1 KB each file... XML would introduce a lot of extra data in overhead.

[frwololo]

Note that some filesystems really don't like a folder with too many files

from StackOverflow:

Performance varies according the the filesystem you're using.

* FAT: forget it (ok, I think the limit is 512 files per directory)
* NTFS: Althought it can hold 4billion files per folder, it degrades relatively quickly - around a thousand you will start to notice performance issues, several thousand and you'll see explorer appear to hang for quite a while.
*

EXT3: physical limit is 32,000 files, but perf suffers after several thousand files too.
*

EXT4: physical limit is 64,000 files.
* ReiserFS, XFS, JFS, BTRFS: these are the good ones for lots of files in a directory as they're more modern and designed to handle many files (the others were designed back in the days when HDDs were measured in MB not GB). Performance is a lot better for lots of files (along with ext4) as they both use a binary search type algorithm for getting the file you want (the others use a more linear one).

http://serverfault.com/questions/98235/ ... a-from-net


also, opening a file is a very costy operation (compared to reading it). We don't notice this on modern computers, but opening 4000 files to read them is significantly slower than opening and reading a file that contains 4000 cards. Expect forge to become really slower on low-end computers

[Sloth]

The cards starting with L are missing!

[zorglub]

Hi everybody,

I thought about it some time ago : "how could I do if I started from zero" ; here is what I ended with (no code yet) :

Maybe data files per set could be a better choice.
But if your code simply reads all data files under a root directory, organisation of such files is just convention, you could change it later whenever you want without changing code. The only thing is avoiding duplicates in the same set.
For this to work, set reference ("M10" for example) has to be part of each card data.

Consequence (expressed in gatherer terms) : "printed" informations (including artist and ) are duplicated between sets, but you can also store "oracle" informations independently of sets.
How to store oracle infos neither in a sole file nor in 4000+ files ? No solid idea about it, just some thoughts :
- use a database (not as straightforward as text files)
- store oracle info per set (with set release date), and retrieve the fresher oracle info for a card.

Just my 2 cents.

[Rob Cashwalker]

Other than FAT16, we're not getting anywhere close to the limits of number of files in a folder. Trust me, I'm familiar with that problem in my day job. I tell my customers not to fill up folders with thousands of phone recordings.... But they're trying to browse the folder in Explorer at the same time as more recordings are being added. The problem with that is that the file I/O functions in the telecom hardware drivers can't deal with delays. I don't think a single pass through the folder or browsing it will be much of an issue. We currently have 4100+ images in the pics folder, and we don't think much of it.

I did just find a bug that only showed itself after the sudden presence of ".svn" in the folder.... But after fixing it, I don't see any significant difference in load time on my laptop - a 1.6 Ghz Centrino/512MB RAM.

The L cards are up now. I had to break it down - Google was breaking the connections. I left it running overnight, and found L had an error during the upload, so I did it today.

[Chris H.]

I have had a chance to try this new revision. I edited a few of the individual card files and found that they were easy to find. The search menu has eased the transition. We had an initial problem with no text and that has been fixed.

I noticed a problem in the deck editor which resulted in an error exception. I merged a quick fix into the SVN.

r2059

- This prevents people from selecting the "Most recently added" menu option from the "Sort By" menu in the deck editor. I discovered that it was leading to an error exception due to the fact that the cards.txt file is now missing entries for the new cards added today.

`

Code: Select all

    private void setupSortMenu() {
        JMenuItem name = new JMenuItem("Card Name");
        JMenuItem cost = new JMenuItem("Cost");
        JMenuItem color = new JMenuItem("Color");
        JMenuItem type = new JMenuItem("Type");
        JMenuItem stats = new JMenuItem("Power/Toughness");
        JMenuItem rarity = new JMenuItem("Rarity");
        JMenuItem newFirst = new JMenuItem("Most recently added");
       
        JMenu menu = new JMenu("Sort By");
        menu.add(name);
        menu.add(cost);
        menu.add(color);
        menu.add(type);
        menu.add(stats);
        menu.add(rarity);
       
        // menu.add(newFirst);
        /*
         *  The "Most recently added" menu now causes an error exception. This will prevent
         *  this option from appearing in the "Sort By" menu. There may be a way (or not)
         *  to sort the indivudual card files by the date created or the date last modified.
         */
       
        this.add(menu);

[Rob Cashwalker]

There may be a chance that at some point I could adapt that function to look at the file date/time info....

[lord of 13]

Is it possible to keep some sort of Dictionary<string, Card> in the class, then load from a file if the card is needed & its name [the key] isn't in the dictionary? While loading, you can use FileInfo to query the time last modified and compare it to the FileInfo of the application itself.

[Rob Cashwalker]

Not without some significant change... I wasn't aware that it would keep hitting the disk other than first-load.

[lord of 13]

Hmm... the dictionary is what I'm implementing in Mindgames, because of how much time it saves at load time. I'm actually using XML, so it makes a HUGE difference in speed.

Significant change... how long would it take, roughly?

[Rob Cashwalker]

I dunno.... Are you thinking of some sort of way to simply persist the data between execution?

[lord of 13]

I would love to, but I can't. If Mindgames enforces some 1K cards [as I believe it might within 6 months], the time taken to load each card & card image [given a conservative load time of about 1/25 of a second] would be 40 seconds. According to good game design theory, making the user wait more than 10 seconds [plus time for other assets!] is bad practice unless you show a progress bar - which I don't want to do. Therefore, it will load the cards on a as-need basis, and persist the cards between modes. However, there will be no persistence between executions [because that would be messy and difficult]. This way, if you play mostly the RPG mode [encouraged], you won't wait 40 seconds - more like 4 seconds, and maybe another second or two per deck you play against.

[Rob Cashwalker]

Well, then we're in the same boat. These card files (I believe) are only read once on startup. CardFactory does seem to make more cards than I think it needs to, but there aren't huge load times because of it.

Card pictures are loaded and unloaded as necessary though.

A Splash Screen with progress bar would be a cool addition of course....