SQLite Data Engine

[Krizzen]

I'd like to propose a possible storage engine for the massive number of cards this project has to face on a daily basis:

http://www.zentus.com/sqlitejdbc/

Its pretty simple to get up and going. It runs native SQLite engines.

Maybe I'm partial to SQL, but I think it'd be cool to "select * from cards where power > 3 and toughness > 3 and text like '%flying%'"

[Snacko]

H2 database http://www.h2database.com/html/main.html is a bit more feature rich than sqlite if you don't need compatibility with sqlite dbs.

[Krizzen]

Oh wow, H2 seems very nice! Few projects boast a small footprint and great performance.

As far as implementing such a beast, I've sifted through the code to try to discover how the cards are loaded, stored in memory, then sorted, searched etc. seemingly to no avail. So...

If a database engine into Forge is feasible, I might can be of assistance. So, if it is possible, any pointers to where I can start looking in the code where I can plug an engine in?

[zerker2000]

Cards are stored as forge.Card objects, in heap. To 'plug an engine in', one would remove all/most variables (except uniqueNumber) from that class, and point the getter/setter methods to the database.

[zerker2000]

Cards are stored as Card objects, in heap. To 'plug an engine in', one would remove all/most variables (except uniqueNumber) from that class, and point the getter/setter methods to the database.

EDIT:They're loaded in CardFactory.getAllCards, executed at startup. They're grouped/sorted/searched mostly with CardList, which is a wrapper for ArrayList<Card>;

[Krizzen]

For some reason I didn't bother to look at CardFactory! Thanks very much!

[Rob Cashwalker]

There's a lot more to a Card object than just text and numbers, which is where the database could come in handy. The Card object contains ability objects which provide functionality, those abilities are generated during runtime during loading. Those objects couldn't be stored in a database.

Use of a database would streamline display of the deck editor and its filtering features. However it would still need to be populated somewhat dynamically, since the card.txt file can be modified at anytime, and we don't want to maintain a single file with all cards.

[zerker2000]

As long as AF abilities include descriptions, would it be feasible to make a "HardcodedAbility", and refactor the cardFactories to use ability text rather than card name? That way, we could load the entire cardsfolder as text for deck editing, but only generate Card objects when we need them (and then store them in a table to avoid redundancy, but it would probably decrease footprint tremendously either way).

[Braids]

Cards are stored as forge.Card objects, in heap. To 'plug an engine in', one would remove all/most variables (except uniqueNumber) from that class, and point the getter/setter methods to the database.

i don't think that is the right approach. we still need Card instances in memory so that they can be differentiated from one another. for example, one card {permanent} has counters on it, and the other does not.

the main features of a relational database are to provide and edit information that is shared among multiple clients, and to provide a powerful query language to as few as one client. i do not think it wise to try to replace Card instances with rows in a table. we have to make them instances at some point in order to manipulate them in the java programming language. pulling them in and out of a database row would add unnecessary complexity and would hamper performance.

i can't think of a good reason why we would be interested in feature . i don't think we want to centralize all oracle card information to a cloud of relational database systems for access by multiple clients. it's easier just to release new batches of card txt files with forge upgrade.

reason would be interesting. we would have the only mtg relational database i know of.

Code: Select all

select name from all_cards
where text like "%enters the battlefield%" and (color = "W" or color = "B");


but how many of our users know how to write sql queries like this? this sounds like it would be more suited as a (very handy) separate web-based service, rather than part of forge itself.

There's a lot more to a Card object than just text and numbers, which is where the database could come in handy. The Card object contains ability objects which provide functionality, those abilities are generated during runtime during loading. Those objects couldn't be stored in a database.

yes. i agree.

Use of a database would streamline display of the deck editor and its filtering features. However it would still need to be populated somewhat dynamically, since the card.txt file can be modified at anytime, and we don't want to maintain a single file with all cards.

i don't see much of a point in keeping both a database and a set of txt files around.

the deck editor's filters could be streamlined by moving from CardLists to Generators. if you want proof, try looking up a card by a substring in its name using the Filter menu in the deck editor. it is Blightning fast. beware, it may leave your deck editor in an unresettable state. i would have converted more CardList uses to Generator in there, but the code was very entwined with the CardList implementation in its fields, etc.

i do understand that a deck editor needs to slurp a generator into a CardList at some point in order to be able to do sorting. but if we can apply as many filters as possible before doing that, it would really speed things up.

Edit: as marked.

[Braids]

As long as AF abilities include descriptions, would it be feasible to make a "HardcodedAbility", and refactor the cardFactories to use ability text rather than card name? That way, we could load the entire cardsfolder as text for deck editing, but only generate Card objects when we need them (and then store them in a table to avoid redundancy, but it would probably decrease footprint tremendously either way).

i tried to do something like this when i implemented the lazy CardFactory.

the main problem is that CardFactory has some insane magick Sleight of Mind inside it. it actually keeps two databases of card instances. one set (forge.CardFactory.allCards) has all of the abilitiies attached, and are owned by the HumanPlayer. the other set (forge.CardFactory.map) has no abilities and no owner, but its contents serve as templates for the first set. not understanding these nuances is probably what caused my lazy CardFactory effort to fail.

Edit: as marked.

[zerker2000]

Well yes, and if we wanted to axe the loading of every card at runtime, we'd have to get rid of both, and move readCards to the deck editor.

[Braids]

Well yes, and if we wanted to axe the loading of every card at runtime, we'd have to get rid of both, and move readCards to the deck editor.

deferring the loading of all cards until they were actually needed is in reality not that helpful. just about everything except playing constructed matches with specific or random (but not randomly generated) decks requires all cards to be loaded into memory. so for standard Forge usage, there is little point.

however, for writing unit tests, it would make test execution much faster. i intend to write a separate CardFactory that lazily loads cards for this very purpose.

edit: minor rewording.

[zerker2000]

Right, except we don't actually need "fleshed out" cards anywhere except play, do we? Also, looking through the code, I don't see any abilities/owner being added to allCards cards. Is that actually the case?
As for unit tests, moving readCards anywhere outside the CardFactory constructor should work, at least if we make a static AllZone.ReadCard(why isn't that class named CardReader, by the way?), and remember to call it if we do need to.

[Braids]

Right, except we don't actually need "fleshed out" cards anywhere except play, do we?

`
you're almost right about "fleshed out" cards only being needed in games. not in play which is an old synonym for on the battlefield. at other times, we just need their oracle/printed attributes. i now see that differentiating between these two uses of the term {card} is very important. this could explain why CardFactory contains two lists. nice find!

Also, looking through the code, I don't see any abilities/owner being added to allCards cards. Is that actually the case?

`
src/forge/CardFactory.java, lines 104-107:

Code: Select all

            while (it.hasNext()) {
                c = getCard(it.next().toString(), AllZone.getHumanPlayer());
                allCards.add(c);
            }


the javadoc for getCard clearly states that it sets the card's owner. you'll have to take it on faith that it also adds abilities.

As for unit tests, moving readCards anywhere outside the CardFactory constructor should work, at least if we make a static AllZone.ReadCard(why isn't that class named CardReader, by the way?), and remember to call it if we do need to.

`
i don't want to move readCards. i prefer to write a separate LazyCardFactory class. the last time i tried to edit CardFactory to make it lazy, i broke Forge in unexpected and quizzical ways.

yes, ReadCard should be named CardReader, but i didn't feel like refactoring it when i was last modifying it.

the depth of your analysis begs the question. are you volunteering for any of this?

edit: formatting changes.

[zerker2000]

Ah, they were being passed through getCard. Do we actually need to do that? The way I see it, the only thing needed outside the game(and that may exclude libraries) is oracle text(and other characteristics), and that should be parseable in ReadCard using Description tags. Even if we don't want to create a separate UnparsedCard class, I'm fairly certain that could be added to Card as a boolean, and checked in e.g. DefaultPlayerZone.add.
By "fleshed out" cards I meant Card Objects, in the MTG Rulebook sense, but yes, the existence of hand/grave abilities does require slightly earlier parsing.
And finally, I'm not quite sure if I am volunteering for anything, but I may indeed have a decent amount of free time within the next month.