MTG Gatherer Downloader

[woogerboy21]

A verify tick would be interesting, but I don't think its needed. Because it would cause a user to be downloading for a very long time, because it would be re-downloading images he or she already have and then replacing them. This could also increase the amount of image corruptions as well as increase the server load. So I wouldn't bother with that option, because people who experiment like me, will click it not knowing what it does. Just to see what it does.

Nah, My thought on the verify option would be simply download the set information and check to see if the image existed in a folder and write any missing files out to a text file. The app already checks to see if the image exists before it downloads it. Infact also checks to see if its there afterwords to. With the verify option you could simply get a listing of images that are missing.

edit:
that would fix your unknown issue, i think

[Godleydemon]

A verify tick would be interesting, but I don't think its needed. Because it would cause a user to be downloading for a very long time, because it would be re-downloading images he or she already have and then replacing them. This could also increase the amount of image corruptions as well as increase the server load. So I wouldn't bother with that option, because people who experiment like me, will click it not knowing what it does. Just to see what it does.

Nah, My thought on the verify option would be simply download the set information and check to see if the image existed in a folder and write any missing files out to a text file. The app already checks to see if the image exists before it downloads it. Infact also checks to see if its there afterwords to. With the verify option you could simply get a listing of images that are missing.

edit:
that would fix your unknown issue, i think

well I still think we need a better gui, lol. something a little more eye catching and more organized.

[Huggybaby]

MWSPA had an sfv file with hashes of all the pics. If a hash didn't match or was missing then the pic was downloaded.

[woogerboy21]

well I still think we need a better gui, lol. something a little more eye catching and more organized.

yea everyone keeps saying this, must be a sign

MWSPA had an sfv file with hashes of all the pics. If a hash didn't match or was missing then the pic was downloaded.

That would work but constantly updating that sounds painfull (may not have been dunno). I was thinking somethimg more simplistic like an md5 sum file located in the folder that stores the images. a small txt file wouldnt be hard to redownload each time you process a set and creating those is one simple command.

[woogerboy21]

I thought so but i wasn't sure.

Thanks for the quick reply! Great work!

This does make me wonder if all the cards that don't have pictures download the same multiverseid. This probably is the case. If so I can work in something that checks to see if the multiverseid is the card backing and warn about the images not being available rather than download useless back images. Hmm... something on my todo list

Well bad news, looks like the gatherer doesnt have a specific multiverse id for just the background images and all card images that isnt available appears as the backing. So no way for me to easily identify if just backings are being downloaded. This option might land on the backburner for the time being.

[Huggybaby]

All the backings would have the exact same CRC32, etc. which is one way to tell.

[Jonix]

I thought so but i wasn't sure.

Thanks for the quick reply! Great work!

This does make me wonder if all the cards that don't have pictures download the same multiverseid. This probably is the case. If so I can work in something that checks to see if the multiverseid is the card backing and warn about the images not being available rather than download useless back images. Hmm... something on my todo list

You may check for checksum of the downloaded image and if it equals the checksum of the backing image (I believe they use the same image in all cases) download the English one.

If you don't have a possibility to check the CRC, you may use the filesize (the backing is 73739 bytes, and normal card images are usually around 20-40 Kb), though this method is much less perfect, since ther is still a possibility that a card might have the same file size as the backing.

[woogerboy21]

Yea generating a file checksum after downloading would let me know but would also add time to the overall process. Think its worth the time overhead?

And the filesize is an option but I was afraid of the same thing, false positive. Have to try different methods.

[Jonix]

Yea generating a file checksum after downloading would let me know but would also add time to the overall process. Think its worth the time overhead?

And the filesize is an option but I was afraid of the same thing, false positive. Have to try different methods.

And optional setting maybe?

[Godleydemon]

When I woke up this morning I had a slight appifany. I wouldn't call it genius, because I have no idea how it would work and I have no idea how to carry it out specifically. Because the scope of the project is huge.

using the program jdownloader or something like that, we can use that as a frame work. It downloads everything from places like media fire without having the pro feature. what about programming a program that utilizes a function like Jdownloader in conjunction with a program that would be like gatherer or mwspa.

for example, we'll call this program, Lore. Lore calls to the picture directory, using a database it finds what pictures your data base needs to complete its directory. Lore then calls to media fire and downloads those specific files, then puts them in the appropriate directories. The problem with this, is that Media fire uses .rars. So the program would have to download the .rar's in a temporary folder. then unzip them automatically, sort through the picture list and select only the files that are needed, move the files to where they need to be, then delete the .rar after wards.

This way we still have the functionality of programs like gatherer and mwspa. But we don't have the constant worry of server stress. its doing most of the work on the computer itself to figure out what to do with the files.

What do you guys think? aye or nay? Its deffinantly a possibility. I just don't know how to do it myself, I can program small programs and can help with some of it. But I deffinantly don't know how to do some of this.

[woogerboy21]

Its possible, there would be a couple issue's that come to mind with that. For starters you would need a clearly defined list of links to image rar's. (it would be nice if there was a formated post somewere that could be queried against). Second you would need a database that contains all the card information (gatherer has alot but does not contain all, such as promo's and specialty sets). And the biggest of them all would be the dependancies between the front application and the apps that would need to be called to perform the compressing / uncompressing. Other than that ok

[woogerboy21]

Alright quick little fyi, think I am done with working on the next release and am sending it through the testing stages. I'll let it run all weekend (to generate hashes mainly) and if all goes well expect an update come monday.

[woogerboy21]

Alright so there might be a delay in the next release. Found a problem but this brought up a couple of questions in my mind and I thought I would ask the users of this application what there oppinion is. For starters the initial upcoming release was to include file hashing capabilities (with generated hashes for low quality images). The main reason for hashing was to identify card backings that were being downloaded from the gatherer on card images the gatherer didnt have an image for (mainly languages other than english). In the application I store the hash for the card backing image so the hash files aren't necessarily needed for application operation.

Question 1:

Code: Select all

So would it be advantageous to delay the release to generate the hashes or not?  (the application can be set to generate hashes during execution so users have the ability to generate there own hashes if they want to ever reproduce hashes.  I will only offer english hashes to begin with.)


Also while waiting for hashes to be generated I embarked on the task of working in the HQ image option. This option would download the .rar or .zip file from the mediafire links. How ever it appears that the mediafire links have some type of redirecting that goes on in the background. So when I can the get command it just kind of hangs for a really long time. How ever the application is designed to only download the HQ .rar's and .zips in the event that either the HQ image is not available in the working temp folder (new in the hq capable version) and that the .rar or .zip file does not exist in the working temp folder. If the file exists it simply uses it, and if the file does not exist and the compressed file exists it simply uncompresses the file and uses its contents bypassing all downloading routines. So it is possible for you to download the HQ compressed files yourself and place them in the temp folder causing the application to bypass the attempt to download the HQ .rar's or .zips.

Question 2:

Code: Select all

As a user would you like to wait to see if I can work the bugs out of the downloading routines or would you want the ability to simply add the .rar or .zip file to the temp folder and use HQ images?


Sorry for the rather long post. Let me know.

[oumve]

Hi woogerboy21!
I don't know if this can help you but I'm currently uploading every HQ scan on a skydrive.I don't use archive,but only single cards
Example : Ancient Hellkite M11

http://cid-f177d76750e73dfb.office.live.com/self.aspx/CCGHQ%20Team%20Pictures/CS013%20-%20Magic%202011%20-%20CCGHQ%20MTG%20M11%20fulls%20080410/Ancient%20Hellkite.full.jpg

If you need something to make it easier for you, just let me know

[woogerboy21]

Hi woogerboy21!
I don't know if this can help you but I'm currently uploading every HQ scan on a skydrive.I don't use archive,but only single cards
Example : Ancient Hellkite M11

http://cid-f177d76750e73dfb.office.live.com/self.aspx/CCGHQ%20Team%20Pictures/CS013%20-%20Magic%202011%20-%20CCGHQ%20MTG%20M11%20fulls%20080410/Ancient%20Hellkite.full.jpg

If you need something to make it easier for you, just let me know

How do you organize the sets and images? Also who is the host for this? I may resort to something like this for individual file pulls. I have been able to get the downloading of the compressed files and the propery extraction and file placing writen. The two issue's that I see using HQ images with the current method for downloading the complete set images is 1.) being such a large file the HTA app shows "not responding" during a download, it still is working the settings for IE cause this. And 2.) I've read online that mediafire's links are only available for a short time period unless you first goto there link they initially post so if I create a URL listing of sets to compressed files there may be the potential for those links to stop working. It would be nice to hear from someone as to if this is true for mediafire.