Deckcheck project -- programmer help wanted

[Marek14]

OK, this is a cross-platform idea. I want to do it for BotArena, but the file formats are similar enough to make it work for Forge as well.

Basically, the idea is to trawl the site http://www.deckcheck.net which has tens of thousands of decks, and add to Forge (or BotArena) those that are supported by the program. I plan to use Erasthotenes sieve algorithm - check for all decks containing a given unimplemented card, moving them away, checking the rest for next card, etc.

There's a hitch, though: in order to do this, I need to download all the decks. It would be best with all of them, as that lets me check them all at once. But there's over 36000 of them, far too many to reasonably click on. This is why I need a help from a programmer. I need an application that would do the following:

1. For all numbers between X and Y, visit the page http://www.deckcheck.net/export.php?deck=<number>&type=txt
2. This leads to a save dialog. Save the file, but prepend the filename with "deckcheck_<number>_" so they would get sorted and there wouldn't be any duplicate names.
3. Save all the files to the same directory.

If someone can make such an application or knows about one that can already do this, I can sieve the decks and find those that will work in Forge. I can even download the new decks and give you new ones when new cards are implemented if you want.

If someone wants to make an automatic sieve, he's certainly welcome to, but it's not required for starters. Getting the files will be enough for now.

Though, in case of Forge, an automatic header-maker would probably be useful as well BotArena requires only minimal editing. Is there a tool that can do "search and replace" automatically in all files within a directory?

[Hellfish]

I've got an old app to massfetch files from websites that I can modify if you don't mind it being a .NET 2.0 app. I can probably toss something together for the sieving and headering too, if you want.

EDIT: Removed the program again already, haha. Gonna see about inserting the header info while fetching, as well as removing an annoying message box I used for debugging.

[Marek14]

That would be great, thanks

As for headering... I don't know much about how the rules are in Forge, but for BotArena I wanted something like this:

// Standard
// [SoA_M10_Zen]

at the beginning of the file, where first line would be the format, and the second the subformat, as they are shown on the deckcheck site. That's optional, however.

What would be very useful is if the line "Sideboard" in the file got a # prepended:

#Sideboard

While BotArena doesn't support sideboarding yet, it supports sideboard section in the deck files

For Forge files, I guess the first line should be the name of the deck (identical to filename), second line "[general]", third "constructed", fourth "[main], and then the "Sideboard" string replaced with "[sideboard]" Someone from the Forge team can say if that's correct

[Hellfish]

Can do, give me a day or so

[Marek14]

Thanks

[Hellfish]

So, while I've got BotArena decks looking right(Not in the "beta" so I can't test), Forge doesn't like my files :/ The attached deck is what is output by my program and the deck editor successfully imports it. However, the deck is nowhere to be found afterwards. Is it the fact that the sideboard is not expected to contain anything?Also, some decks don't have info pages so they won't get a name.

[Marek14]

As I said, I don't know about Forge format too well, I just asked here to get a better chance So I can't help with this too much.

This is an example of a file importable to BotArena:

deckcheck_01415_fourcolorgreen.txt

// Standard - [Mrd_8th_SoK]
1 Minamo, School at Water's Edge
1 Okina, Temple to the Grandfathers
2 Mountain
2 City of Brass
4 Blinkmoth Nexus
3 Island
10 Forest
1 Plains
1 Meloku the Clouded Mirror
2 Eternal Witness
3 Trinket Mage
3 Etched Oracle
4 Sakura-Tribe Elder
4 Birds of Paradise
3 Naturalize
2 Condescend
2 Mana Leak
2 Fireball
2 Engineered Explosives
1 Sensei's Divining Top
1 Sword of Light and Shadow
4 Sword of Fire and Ice
2 Kodama's Reach

#Sideboard
4 Plow Under
4 Sowing Salt
3 Circle of Protection: Red
3 Boil
1 Engineered Explosives

Of course, the format changes considerably once BotArena saves it (it stores percentages of various card types etc.) But it can open this, and that's important.

I'm not sure what you mean by the lack of info pages. Each deck has a filename under which it's normally saved. It's this filename that I consider the deck name. I noticed that some decks have japanese/chinese characters for a name, I'm not quite sure under what name THEY would get saved -- not tested this.

[Hellfish]

Hmm.. Turns out there is a bug(or misfeature, who can tell anymore? ;P) in .NET. Here's a version that will save into the BotArena format only and without fetching the format/set/deck name. The other version, that encountered said bug, could fetch the deck with name,format and set but it would invariably hang when fetching more than 1 deck at a time, which is pretty damn useless. I will attempt a rewrite of that version this weekend.

[Marek14]

OK, I'll have a look

Just a quick question -- where exactly are the decks saved?

EDIT: Never mind, I found them Though, there's one more thing (although just a minor one): the deck number should be filled with 0's from the left to always have five digits for better sorting But that can be taken care of with mass renaming, so it's not a big problem.

[Marek14]

OK, I downloaded all the decks and started with the sieve algorithm. Here is the first fully implemented BotArena deck I found.

This is probably the final version of the header. Not sure if it could be automatized. The rest of deck is just the format BotArena re-saves them in.

deckcheck_00018_affinity.txt

// Affinity
// Jack Calvi
// Extended
// Tp_6th_ChK
// NG New Years Day
// 01. Jan 2005
// 5th place
// Total cards in deck - 60

// Creatures - 25 (41.67%)

4 Arcbound Ravager (15300)
4 Arcbound Worker (15304)
4 Disciple of the Vault (14902)
4 Frogmite (15007)
3 Meddling Mage (12804)
3 Myr Enforcer (15010)
3 Somber Hoverguard (15009)

// Sorceries - 4 (6.67%)

4 Thoughtcast (15014)

// Artifacts - 7 (11.67%)

4 Chromatic Sphere (14873)
3 Cranial Plating (15458)

// Lands - 20 (33.33%)

2 Ancient Den (14926)
2 Blinkmoth Nexus (15249)
2 City of Brass (1523)
4 Darksteel Citadel (15282)
2 Glimmervoid (14928)
4 Seat of the Synod (14930)
4 Vault of Whispers (14932)

#SIDEBOARD

// Total cards in sideboard - 15

// Creatures

3 Kami of Ancient Law (15659)

// Instants

Oxidize (15201)

// Sorceries

3 Cabal Therapy (13867)

// Enchantments

3 Chill (587)
3 Engineered Plague (1042)
2 Seal of Removal (12068)

[Hellfish]

Sorry it took a while to respond but I'm away from my dev box right now so I can't do much either way. That header *could* be automatically constructed, unless that .NET bug refuses to cooperate. And even then, only for the decks that have full info pages.

[Marek14]

For now I have downloaded everything and I've implemented several hundred decks in BotArena. I'm still not even half-way there.

[mtgrares]

If I'm understanding the conversation, you have a bunch of BotArena decks and you what them converted into Forge style decks?

p.s. I just type BatArena instead of BotArea, ha.

[Marek14]

Well, the decks downloaded through HellFish's program are pretty basic text files, I just asked to convert the "Sideboard" string into "#Sideboard", and then BotArena is able to load them. However, after they are loaded, I still have to do some work, mostly changing the images (no point in having Lorwyn decks with M11 Garruk Wildspeaker).

Forge, I think, uses a more complicated header structure (or, rather, it uses a header structure, period).

If you like, try the application (or download some decks manually), and look what exactly is needed to load them.