Gatherer Downloader App (v1.6.8) [hq pics / card database]

[woogerboy21]

Also one other question that just came to mind. Is there any particular reason you create a folder named "fulls" when downloading the full card images for the magic album configuration (and probably wagic as well)? The "fulls" and "crops" folder names are not used in those applications and if its not to much trouble to have them not created when those two applciation sets are loaded that would be great!

[Ark]

The base folder name is Origs, Fulls, or Crops, to ensure that if you select 2 or more of those options, that they are able to go into different locations. I don't think any application uses those folder names, not even MWS.
I suppose I could add an option to not create that folder, but then make it so you can't select more than 1 of those file types from the checkboxes.

[woogerboy21]

Its up to you, I was just trying to eleminate as many steps outside the application as possible to move the images into the expected applications image folder. It not that hard to simply move the folders.

[woogerboy21]

I may have found another issue. (Annoyed with me yet?)
So far if I download all the card database information starting from Alpha and then download the card images I everything works ok. But if I download all the card data with "Additional Card Information", when I attempt to download the card images when it makes it to the card "Will-o-the-wisp" (didn't check the set version) I get errors connecting to gatherer and am never able to get anymore images past that card. Even though I can get to the gatherer and the application works fine without the additional card information.

Can you reproduce the problem on your side to see if you get the same results?

[Ark]

What set, and can I download just that 1 set with additional information to try to reproduce it? Downloading every single set with all card info will take a long time.
Also, if you save the database, close, restart the app, then reload the same database with all sets and all additional card information, does the problem still happen?

[woogerboy21]

Maybe it was just a fluke. I cant seem to reproduce the problem. After saving and reloading the data everything has downloaded alright. I'll see if I can reproduce the behavior in a single set.

[Ark]

v1.3.0 has been posted.

Fixed bug with blank entries not counting as missing in the CSV set/symbol lists (This makes v1.2.x miss a lot of the set subfolders)
Added debug info (created in GathererDownloader.txt) for when an image download from wizards is aborted
Added option to not use per-format folders (if only using 1 format)
Fixed card image saving for originals to actually save byte for byte the original instead of load/recompress JPG. This is how mana/set symbols always worked.
Added option to save planes/schemes as original size when saving fulls
Added option to download HQ fulls from ark42.com
Added work-around for gatherer listing some split cards as having alternate art
Added work-around for BFM and Erase (Not the Urza's Legacy One) name formatting
Added work-around for cards listed in the text spoiler that aren't actually in the set (Sea Eagle in 9th)

Tweaked MWSHQ images on my server to match formatting needed for downloading (Such as AEther -> Aether, Forest1a -> Forest5, Kill! Destroy! -> Kill Destroy, Ertai, the Corrupted -> Ertai, the Corrupted1), and uploaded the duel decks/from the vaults/etc.
I tested a full run of downloading all 3.06GB of 18,067 files to make sure they all are there.

If you get into the situation where the program can't connect to the website over and over again, it should create the debug text file after you give up. Please send me the contents of that file (minus the @@@ filler at the end).

[woogerboy21]

Wow, nice update! Decided to host the HQ images youself huh? Good luck with that. If you decide people pulling the images gets to much for your available bandwidth you might consider allowing someone to define were the HQ images are pulled from. (Just a thought). Going to try the updated version now.

[Ark]

Well it's not like anybody else couldn't get the images I've had on ark42.com forever anyway. They're used on all my other pages like card search and pricelist. It would take a bunch of people all at once to really kill the server, and I get 6000GB/month transfer on that server. It's mainly for my actual company that sells software though: http://www.morpheussoftware.net/

It wouldn't really be possible to let the user define the path, because you can't just sub in variables like set name, card name, and/or multiverse ID number to the URLs. You have to do transformations like map Æ to Ae on my server, but you might not do that on others, depending on how people host the files.
Plus, you have to have ALL cards from all sets, including the random things like the tokens from some of the duel decks, because Gatherer happens to list them as cards. Otherwise, the download will stop a lot to notify you of missing cards or bad URLs.

[woogerboy21]

Understood you would have to replicate the image name and folder structure on the host that houses the image files for the application, it was just a thought to offer others the ability to help host the images as well as if you decide to ever stop hosting the images. Though I don't have as fast an upload speed as your host does I would offer to help host the images as well if you ever wanted an alternate site.

Infact to replicate the expected structure wouldn't you just need to run the application and download all the images using one of the settings (don't know which one) that the application supports?

[Ark]

The closest thing is to start with the MWS defaults, but use the full set name instead of the set code for the folder.
Except then, you have to rename any cards with non-ASCII letters in their names, because even though my server has Dandan.full.jpg or Aether Adept.full.jpg, it will save onto your harddrive as Dandân.full.jpg or Æther Adept.full.jpg because that is what the Name column from Gatherer will show.
You might also have to change some (but not all) of the token cards from things like Goblin token card.full.jpg to just Goblin.full.jpg. I think the UN-sets on Gatherer actually say "token card" in the name, but the files on my server don't. And of course, MOST sets don't have tokens in Gatherer anyway, so you only have a handful of random tokens mixed in with the cards just to appease the downloads.

[zeka711]

great tool,

if language is set to german, many sets worked problem-free.
but shadowmoor: Gloomwidow's Feast, caused a "XML parse error"

[Ark]

Yeah, it looks like Gatherer has the following invalid markup:

<div class="cardtextbox">Zerstöre eine fliegende Kreatur deiner Wahl. Falls diese Kreatur blau oder schwarz war, bringe einen 1/2 grünen Spinnen-Kreaturenspielstein mit Reichweite ins Spiel. <i>(Er kann fliegende Kreaturen</div><div class="cardtextbox"> blocken.)</i></div></div>

(the <i> tag starts, then the </div> tag closes and another <div> tag opens, with the ending </i> inside a different div. Clearly an idiotic thing to do)


Not sure why they did that or why no English card has that bug, but it is again another bug thanks to the programmers at Wizards generating invalid XML. I'll see if I can make a work-around for it in the next release.

[Ark]

v1.3.1 is posted.

I added a work-around to remove that div close/reopen in the middle of an <i> tag so that the XML will parse properly on those cards.
I fixed the Load Defaults button in the Image Formats window so that if you press Cancel, it actually goes back to what you had before you went into the Image Formats window, instead of back to the last defaults you loaded.
I added an additional Gatherer string fix for multiple hybrid mana symbols like Foxfire Oak and Figure of Destiny showing up in the text spoiler with the missing } symbol.
Finally, I improved the debug info that will be created in GathererDownloader.txt when you pick No to continue trying to download images after it tells you it can't connect to the website. If you cancel/restart downloading a lot and this happens to you, please send me this text file so I can look into it.

[Ark]

v1.3.2 - I removed the debug text file that is generated, and made 1 slight tweak to the internet code which I hope will eliminate the random times you can't connect anymore until you close the program, although I have a very hard time reproducing it, I've spent hours clicking cancel and restarting various downloads of different sets and images, changing it to a new set and new images to download every few seconds and I haven't gotten it to happen anymore.

If you get the message about not being able to reach the website and clicking 'Yes' to try again just gets you the same message over and over again, AND you *know* you are online still because it just worked a second ago, AND you are using v1.3.2, then please let me know. Especially if you have a way to reproduce the issue faster.