Gatherer Downloader App (v1.6.8) [hq pics / card database]

[Ark]

The file/load operations are not multithreaded: the application just loads the file, parses it and fills in the big table. It is to be expected then that no UI interaction will work until the file is 100% loaded, but there is no case for deadlocks this way either, so the application is guaranteed to always come back to responsive after the file loads.
I think the thing that takes the most time is populating the listview table - the file is loaded and the XML parses very fast, but windows is slow to accept adding the rows to the table all at once. I disable window painting during load, otherwise you have to watch each line be added one by one, and the screen has to refresh for each one. If I allow this to happen, it can take an hour to load the huge XML file, because each new line added has to redraw EVERY previous line and then scroll to the bottom. The first few go fast, but each one takes a bit longer and longer than the last.

I'll try to duplicate the issue with the downloading of all of the sets using the list you posted above.

Btw, could you produce for me a 5-column CSV file for all sets like we mentioned before? Specifically, I need the 4th and 5th columns for what Magic Album needs for the Category column as well as the set name for when it differs. Remember to just omit the 5th column (and 2nd) when it is the same as the 3rd (or 1st).

[Ark]

I think I found the issue -- GDI object handle exhaustion. I might have forgotten to free an HBITMAP handle in the creation of the tiny 22x16 thumbnail image added to the left column as the images are downloaded
Watch the app process in Task Manager (add the GDI Objects column if you don't have it shown). Once GDI Objects reaches 10,000 or so, no new GUI objects can be redrawn, and that's probably what you're seeing. I've fixed it in the source code for the next release, it shouldn't happen anymore, but I'm not going to leave the app running to test. I saw GDI objects hit 1,000 earlier, but now it stays steady for, so I'm sure its good now.

[woogerboy21]

Great! I'll see about getting a csv sets list for you in the morning. One other thing I've noticed and not exactly sure why its happening but for every set that contains multiple images for basic lands, each image version is downloaded and named according to the image options set with the exception of the battle royale set. On the gatherer it shows 9 different image versions for each basic land but only the first version is downloaded. Any ideas why this set wouldnt be downloading all the card image versions?

[Ark]

Yeah, you entered "BRB-Battle Royale Box Set" instead of "BR/BRB-Battle Royale Box Set" for that line.

[woogerboy21]

Is this what you are expecting?

noapp_setlist | Open

Code: Select all

1E,A,Limited Edition Alpha,,
2E,B,Limited Edition Beta,,
2U,U,Unlimited Edition,,
AN,,Arabian Nights,,
AQ,,Antiquities,,
3E,R,Revised Edition,,
LE,LG,Legends,,
DK,,The Dark,,
FE,,Fallen Empires,,
4E,,Fourth Edition,,
IA,,Ice Age,,
CH,,Chronicles,,
HM,HL,Homelands,,
AL,,Alliances,,
MI,,Mirage,,
VI,,Visions,,
5E,,Fifth Edition,,
PO,PT,Portal,,
WL,,Weatherlight,,
TE,,Tempest,,
ST,SH,Stronghold,,
EX,,Exodus,,
P2,,Portal Second Age,,
UG,,Unglued,,
UZ,US,Urza's Saga,,
GU,UL,Urza's Legacy,,
6E,,Classic Sixth Edition,,
CG,UD,Urza's Destiny,,
PK,P3,Portal Three Kingdoms,,
P3,ST,Starter 1999,,
MM,,Mercadian Masques,,
BR,BRB,Battle Royale Box Set,,
NE,,Nemesis,,
P4,S2K,Starter 2000,,
PR,PY,Prophecy,,
IN,,Invasion,,
BD,,Beatdown Box Set,,
PS,,Planeshift,,
7E,,Seventh Edition,,
AP,,Apocalypse,,
OD,,Odyssey,,
TOR,TO,Torment,,
JUD,JU,Judgment,,
ONS,ON,Onslaught,,
LGN,LE,Legions,,
SCG,SC,Scourge,,
8ED,8E,Eighth Edition,,
MRD,MR,Mirrodin,,
DST,DS,Darksteel,,
5DN,FD,Fifth Dawn,,
CHK,,Champions of Kamigawa,,
UNH,,Unhinged,,
BOK,,Betrayers of Kamigawa,,
SOK,,Saviors of Kamigawa,,
9ED,9E,Ninth Edition,,
RAV,,Ravnica: City of Guilds,,
GPT,GP,Guildpact,,
DIS,,Dissension,,
CSP,CS,Coldsnap,,
TSP,,Time Spiral,,
TSB,,Time Spiral "Timeshifted",,
PLC,,Planar Chaos,,
FUT,,Future Sight,,
10E,,Tenth Edition,,
LRW,,Lorwyn,,
EVG,,Duel Decks: Elves vs. Goblins,,
MOR,,Morningtide,,
SHM,,Shadowmoor,,
EVE,,Eventide,,
DRB,,From the Vault: Dragons,,
ALA,,Shards of Alara,,
DD2,,Duel Decks: Jace vs. Chandra,,
CON,CFX,Conflux,,
DDC,,Duel Decks: Divine vs. Demonic,,
ARB,,Alara Reborn,,
M10,,Magic 2010,,
V09,,From the Vault: Exiled,,
HOP,,Planechase,,
ZEN,,Zendikar,,
DDD,,Duel Decks: Garruk vs. Liliana,,
H09,,Premium Deck Series: Slivers,,
WWK,,Worldwake,,
DDE,,Duel Decks: Phyrexia vs. the Coalition,,
ROE,,Rise of the Eldrazi,,
ARC,,Archenemy,,
M11,,Magic 2011,,
V10,,From the Vault: Relics,,
DDF,,Duel Decks: Elspeth vs. Tezzeret,,
SOM,,Scars of Mirrodin,,
PD2,,Premium Deck Series: Fire and Lightning,,
MBS,,Mirrodin Besieged,,


ma_setlist | Open

Code: Select all

1E,A,Limited Edition Alpha,Core Sets,1st Limited Edition Alpha
2E,B,Limited Edition Beta,Core Sets,1st Limited Edition Beta
2U,U,Unlimited Edition,Core Sets,2nd Unlimited
AN,AN,Arabian Nights,Expansions,Arabian Nights
AQ,AQ,Antiquities,Expansions,Antiquities
3E,R,Revised Edition,Core Sets,3rd Revised Edition
LE,LG,Legends,Expansions,Legends
DK,DK,The Dark,Expansions,The Dark
FE,FE,Fallen Empires,Expansions,Fallen Empires
4E,4E,Fourth Edition,Core Sets,4th Edition
IA,IA,Ice Age,Expansions,Ice Age
CH,CH,Chronicles,Special Sets,Chronicles
HM,HL,Homelands,Expansions,Homelands
AL,AL,Alliances,Expansions,Alliances
MI,MI,Mirage,Expansions,Mirage
VI,VI,Visions,Expansions,Visions
5E,5E,Fifth Edition,Core Sets,5th Edition
PO,PT,Portal,Special Sets,Portal
WL,WL,Weatherlight,Expansions,Weatherlight
TE,TE,Tempest,Expansions,Tempest
ST,SH,Stronghold,Expansions,Stronghold
EX,EX,Exodus,Expansions,Exodus
P2,P2,Portal Second Age,Special Sets,Portal Second Age
UG,UG,Unglued,Special Sets,Unglued
UZ,US,Urza's Saga,Expansions,Urza's Saga
GU,UL,Urza's Legacy,Expansions,Urza's Legacy
6E,6E,Classic Sixth Edition,Core Sets,6th Classic Edition
CG,UD,Urza's Destiny,Expansions,Urza's Destiny
PK,P3,Portal Three Kingdoms,Special Sets,Portal Three Kingdoms
P3,ST,Starter 1999,Special Sets,Starter 1999
MM,MM,Mercadian Masques,Expansions,Mercadian Masques
BR,BRB,Battle Royale Box Set,Special Sets,Battle Royale
NE,NE,Nemesis,Expansions,Nemesis
P4,S00,Starter 2000,Special Sets,Starter 2000
PR,PY,Prophecy,Expansions,Prophecy
IN,IN,Invasion,Expansions,Invasion
BD,BD,Beatdown Box Set,Special Sets,Beatdown
PS,PS,Planeshift,Expansions,Planeshift
7E,7E,Seventh Edition,Core Sets,7th Edition
AP,AP,Apocalypse,Expansions,Apocalypse
OD,OD,Odyssey,Expansions,Odyssey
TOR,TO,Torment,Expansions,Torment
JUD,JU,Judgment,Expansions,Judgment
ONS,ON,Onslaught,Expansions,Onslaught
LGN,LE,Legions,Expansions,Legions
SCG,SC,Scourge,Expansions,Scourge
8ED,8E,Eighth Edition,Core Sets,8th Edition
MRD,MR,Mirrodin,Expansions,Mirrodin
DST,DS,Darksteel,Expansions,Darksteel
5DN,FD,Fifth Dawn,Expansions,Fifth Dawn
CHK,CHK,Champions of Kamigawa,Expansions,Champions of Kamigawa
UNH,UNH,Unhinged,Special Sets,Unhinged
BOK,BOK,Betrayers of Kamigawa,Expansions,Betrayers of Kamigawa
SOK,SOK,Saviors of Kamigawa,Expansions,Saviors of Kamigawa
9ED,9E,Ninth Edition,Core Sets,9th Edition
RAV,RAV,Ravnica: City of Guilds,Expansions,Ravnica City of Guilds
GPT,GP,Guildpact,Expansions,Guildpact
DIS,DIS,Dissension,Expansions,Dissension
CSP,CS,Coldsnap,Expansions,Coldsnap
TSP,TSP,Time Spiral,Expansions,Time Spiral
TSB,TSB,Time Spiral "Timeshifted",Expansions,Time Spiral Timeshifted
PLC,PLC,Planar Chaos,Expansions,Planar Chaos
FUT,FUT,Future Sight,Expansions,Future Sight
10E,10E,Tenth Edition,Core Sets,10th Edition
LRW,LRW,Lorwyn,Expansions,Lorwyn
EVG,EVG,Duel Decks: Elves vs. Goblins,Special Sets,Duel Decks - Elves vs. Goblins
MOR,MOR,Morningtide,Expansions,Morningtide
SHM,SHM,Shadowmoor,Expansions,Shadowmoor
EVE,EVE,Eventide,Expansions,Eventide
DRB,DRB,From the Vault: Dragons,Special Sets,From the Vault - Dragons
ALA,ALA,Shards of Alara,Expansions,Shards of Alara
DD2,DD2,Duel Decks: Jace vs. Chandra,Special Sets,Duel Decks - Jace vs. Chandra
CON,CFX,Conflux,Expansions,Conflux
DDC,DDC,Duel Decks: Divine vs. Demonic,Special Sets,Duel Decks - Divine vs. Demonic
ARB,ARB,Alara Reborn,Expansions,Alara Reborn
M10,M10,Magic 2010,Core Sets,Magic 2010
V09,V09,From the Vault: Exiled,Special Sets,From the Vault - Exiled
HOP,HOP,Planechase,Special Sets,Planechase
ZEN,ZEN,Zendikar,Expansions,Zendikar
DDD,DDD,Duel Decks: Garruk vs. Liliana,Special Sets,Duel Decks - Garruk vs. Liliana
H09,H09,Premium Deck Series: Slivers,Special Sets,Premium Deck Series - Slivers
WWK,WWK,Worldwake,Expansions,Worldwake
DDE,DDE,Duel Decks: Phyrexia vs. the Coalition,Special Sets,Duel Decks - Phyrexia vs. The Coalition
ROE,ROE,Rise of the Eldrazi,Expansions,Rise of the Eldrazi
ARC,ARC,Archenemy,Special Sets,Archenemy
M11,M11,Magic 2011,Core Sets,Magic 2011
V10,V10,From the Vault: Relics,Special Sets,From the Vault - Relics
DDF,DDF,Duel Decks: Elspeth vs. Tezzeret,Special Sets,Duel Decks - Elspeth vs. Tezzeret
SOM,SOM,Scars of Mirrodin,Expansions,Scars of Mirrodin
PD2,PD2,Premium Deck Series: Fire and Lightning,Special Sets,Premium Deck Series - Fire & Lightning
MBS,MBS,Mirrodin Besieged,Expansions,Mirrodin Besieged


[Ark]

A mix of the two is what I need, although the 2nd would technically work, it is unnecessarily too big.
I need full lines like this for when the values differ:
1E,A,Limited Edition Alpha,Core Sets,1st Limited Edition Alpha

But these type of lines:
AL,AL,Alliances,Expansions,Alliances
6E,6E,Classic Sixth Edition,Core Sets,6th Classic Edition
are better off just:
AL,,Alliances,Expansions
6E,,Classic Sixth Edition,Core Sets,6th Classic Edition

And I'm pretty sure this one:
TSB,TSB,Time Spiral "Timeshifted",Expansions,Time Spiral Timeshifted
must be:
TSB,,"Time Spiral ""Timeshifted""",Expansions

You need to double up quotes in CSV and then quote the whole string. You also need to quote the string if it contains commas or newlines. That will read into memory as Time Spiral "Timeshifted" and since "s are automatically stripped from path names anyway, you don't need the 5th column to be present at all.

I can probably work that 2nd list into what I need though later today.

[Ark]

This is the list I will be including in the future, let me know if you see any issues.

list | Open

Core Sets,1E,Limited Edition Alpha,A,1st Limited Edition Alpha
Core Sets,2E,Limited Edition Beta,B,1st Limited Edition Beta
Core Sets,2U,Unlimited Edition,U,2nd Unlimited
Expansions,AN,Arabian Nights
Expansions,AQ,Antiquities
Core Sets,3E,Revised Edition,R,3rd Revised Edition
Expansions,LE,Legends,LG
Expansions,DK,The Dark
Expansions,FE,Fallen Empires
Core Sets,4E,Fourth Edition,,4th Edition
Expansions,IA,Ice Age
Special Sets,CH,Chronicles
Expansions,HM,Homelands,HL
Expansions,AL,Alliances
Expansions,MI,Mirage
Expansions,VI,Visions
Core Sets,5E,Fifth Edition,,5th Edition
Special Sets,PO,Portal,PT
Expansions,WL,Weatherlight
Expansions,TE,Tempest
Expansions,ST,Stronghold,SH
Expansions,EX,Exodus
Special Sets,P2,Portal Second Age
Special Sets,UG,Unglued
Expansions,UZ,Urza's Saga,US
Expansions,GU,Urza's Legacy,UL
Core Sets,6E,Classic Sixth Edition,,6th Classic Edition
Expansions,CG,Urza's Destiny,UD
Special Sets,PK,Portal Three Kingdoms,P3
Special Sets,P3,Starter 1999,ST
Expansions,MM,Mercadian Masques
Special Sets,BR,Battle Royale Box Set,BRB,Battle Royale
Expansions,NE,Nemesis
Special Sets,P4,Starter 2000,S00
Expansions,PR,Prophecy,PY
Expansions,IN,Invasion
Special Sets,BD,Beatdown Box Set,,Beatdown
Expansions,PS,Planeshift
Core Sets,7E,Seventh Edition,,7th Edition
Expansions,AP,Apocalypse
Expansions,OD,Odyssey
Expansions,TOR,Torment,TO
Expansions,JUD,Judgment,JU
Expansions,ONS,Onslaught,ON
Expansions,LGN,Legions,LE
Expansions,SCG,Scourge,SC
Core Sets,8ED,Eighth Edition,8E,8th Edition
Expansions,MRD,Mirrodin,MR
Expansions,DST,Darksteel,DS
Expansions,5DN,Fifth Dawn,FD
Expansions,CHK,Champions of Kamigawa
Special Sets,UNH,Unhinged
Expansions,BOK,Betrayers of Kamigawa
Expansions,SOK,Saviors of Kamigawa
Core Sets,9ED,Ninth Edition,9E,9th Edition
Expansions,RAV,Ravnica: City of Guilds
Expansions,GPT,Guildpact,GP
Expansions,DIS,Dissension
Expansions,CSP,Coldsnap,CS
Expansions,TSP,Time Spiral
Expansions,TSB,"Time Spiral ""Timeshifted"""
Expansions,PLC,Planar Chaos
Expansions,FUT,Future Sight
Core Sets,10E,Tenth Edition,,10th Edition
Expansions,LRW,Lorwyn
Special Sets,EVG,Duel Decks: Elves vs. Goblins,,Duel Decks - Elves vs. Goblins
Expansions,MOR,Morningtide
Expansions,SHM,Shadowmoor
Expansions,EVE,Eventide
Special Sets,DRB,From the Vault: Dragons,,From the Vault - Dragons
Expansions,ALA,Shards of Alara
Special Sets,DD2,Duel Decks: Jace vs. Chandra,,Duel Decks - Jace vs. Chandra
Expansions,CON,Conflux,CFX
Special Sets,DDC,Duel Decks: Divine vs. Demonic,,Duel Decks - Divine vs. Demonic
Expansions,ARB,Alara Reborn
Core Sets,M10,Magic 2010
Special Sets,V09,From the Vault: Exiled,,From the Vault - Exiled
Special Sets,HOP,Planechase
Expansions,ZEN,Zendikar
Special Sets,DDD,Duel Decks: Garruk vs. Liliana,,Duel Decks - Garruk vs. Liliana
Special Sets,H09,Premium Deck Series: Slivers,,Premium Deck Series - Slivers
Expansions,WWK,Worldwake
Special Sets,DDE,Duel Decks: Phyrexia vs. the Coalition,,Duel Decks - Phyrexia vs. The Coalition
Expansions,ROE,Rise of the Eldrazi
Special Sets,ARC,Archenemy
Core Sets,M11,Magic 2011
Special Sets,V10,From the Vault: Relics,,From the Vault - Relics
Special Sets,DDF,Duel Decks: Elspeth vs. Tezzeret,,Duel Decks - Elspeth vs. Tezzeret
Expansions,SOM,Scars of Mirrodin
Special Sets,PD2,Premium Deck Series: Fire and Lightning,,Premium Deck Series - Fire & Lightning
Expansions,MBS,Mirrodin Besieged

[Ark]

I posted v1.2.0 and it use a 5 column CSV format for the set list now.
WARNING! If you had v1.1.0 or lower installed, your set list and symbol list will be ignored and set back to the new default, because the format has changed.
I added the ability to use the alternate set names for the folders when creating the images, as well as the category sub folders.
I also changed the crop for Fulls to always do 6 pixels away from each side, by checking the image size first. Crops will try to do a special case if the size matches the known-size for Planes or Schemes, in order to crop out just the art, otherwise the crop size will fall back to assuming the the regular card size.
And of course, that bug where you download over ~10,000 images and the UI stops redrawing, should be fixed.

[woogerboy21]

Sweet! Ill give it a shot today and let you know how everything works out. Thanks for working on getting the magic album naming worked into the downloader. Now we just have to work in Wagic

I think most of everything is already in place to support Wagic, should just need a configuration from the drop down list in the image options page. Let me see if I can get you the information for that.

[woogerboy21]

Alright it appears the UI no longer hangs (good job). Also is there a chance you could create a wagic image drop down menu option? The settings should use the same folder names as MWS but the card names should be the images multiverse id. And if you REALLY want to work in the ability to zip all the images up into a non-compressed zip file (per set) wagic also supports that option but that might be out of the scope of this project unless you see fit.

Once my downloads of all the images complete I will try the images out in the magic album application to see if I see any problems but I am not expecting any issues.

edit:
Also one other question, not to steal chaudakh thunder with his application but I thought he had a neat idea pulling higher quality card scans from the magiccards.info website. Thoughts on working this in for the image source? It looks like there URL to card images is consistant and simple to figure out.

http://magiccards.info/scans/{languageabbr}/{setabbr}/{cardnumber}.jpg
Example: http://magiccards.info/scans/en/m11/2.jpg

[Ark]

Let me know the defaults for Wagic and I can add it to the list.
Zip is possible if you really want, but I don't really see the point if it accepts unzipped anyway.

Wow, that application looks suspiciously like mine...
I don't consider magiccards.info a trusted source though. Where does that site get there images from? Do they scan them all themselves, or what? Besides, they aren't as HQ as the downloads you can manually get from here, they are only just slightly bigger than what is on Gatherer. I could pull those HQ ones (for English at least) from my site ark42.com, since I keep them all there in an even easier URL format than magiccards.info does, but I'm not sure I want to handle that much traffic.

[woogerboy21]

Let me know the defaults for Wagic and I can add it to the list.
Zip is possible if you really want, but I don't really see the point if it accepts unzipped anyway.

Which default specifically are you looking for? The folder structure format is the set abbr that MWS uses and the card image names are the MultiverseID with the .jpg extension. And they are full images. What other information do you need? It appears the current sets list should handle the applications images.

I don't consider magiccards.info a trusted source though. Where does that site get there images from? Do they scan them all themselves, or what?

I have no idea what images they use but they are of better quality than the gatherer. I thought I heard (and could be wrong) that magiccards.info used images from here only saved at a slightly lower jpg quality for most card scans they show on there site. Atleast for the english language that is.

I could pull those HQ ones (for English at least) from my site ark42.com, since I keep them all there in an even easier URL format than magiccards.info does, but I'm not sure I want to handle that much traffic.

I was thinking magiccards.info actually has the bandwidth / hardware in place to support people pulling the images from them (since there site is about finding details for magic cards out) and didn't want to burden any one individual with trying to support downloads of the images.

[Ark]

I'm looking for the exact set of options. Just take a screenshot of the Image Formats window how it needs to be set up for Wagic and I'll add it to the dropdown.

[woogerboy21]

Ran into a problem. For whatever reason the "Timespiral Time Shifted" set is not being properly placed inside its catagories folder. Are you able to get the set downloaded into its correct catagory folder? Also here is the screenshot for the wagic image options:

| Open

[Ark]

For some reason, I had a hard-coded "fix" to set names retrieved from gatherer to translate "Timeshifted" into just Timeshifted, and that was later causing the set name not to be found in the set list when it tried to lookup the set code or the category. I've removed that translation, as well as added the defaults for Wagic. v1.2.1 posted.