MTG Gatherer Downloader

[Huggybaby]

I'm short on time now, but yes this looks useful!

[woogerboy21]

I'm short on time now, but yes this looks useful!

There one thought I had when adding the high quality functionality. Allow for the same process the team uses when posting and naming the files in each release. I have compiled a setinfo file that contains links to hq images (full) and most (cropped) for the sets available on the gatherer. However during the process and testing I have noticed there doesnt really appear to be any uniformity to how set images were released through the years. If there has been let me know. An example would be it appears the naming tried to follow a process of name full images with the .full name part and crops with the .crop. And it looks like the file names were targeted for the magic workstation application. But there seems to be errors in card spellings as well as certain scans not included in a set release (like tokens, I understand tokens were put into there own zip but not sure why the decision was made to do it this way). Im not trying to slander im getting to the point. The problem that occurs is when I query a database such as wizards for data they list may be inconsistant with the cards in a particular release. So to code around the difference is difficult. So far I haven't put any special coding into the app to try and code around problems in the releases simply cause I didnt think there should be specialized code for something like a mispelled image name. Do you have any suggestions? Is there any plans to ever go back and uniform the set releases? Other idea's?

The only two real options I can think of is to either go back, choose a single source to query the data and uniform the releases to them (ex: query the gatherer and make sure all the cards listed for the "alpha" set is in the alpha zip on mediafire.) Most sets are probably correct but I've noticed in testing some here and there small problems occur.

OR

Have the app when using the high quality mode default to low if there isnt a card found in high quality. (very prone to simple errors.)

I'm not to fond of the defaulting back to the low quality option personally but I know uniforming the releases would be alot of work. What are your thoughts? I thought about going through a set at a time when users pointed out problems but I don't have any special media fire account and I dont know the details of how the CCGHQ team operates and didn't want to come in barking problems or orders etc.

Please don't take anything I said personally, all the efforts are wonderfull. I would just like to see things organized a bit better and would be willing to help do so.

[Huggybaby]

Nothing taken personally I assure you. Our sets do need a refresh. I just can't find anyone to work with to make it happen. I need someone(s) to be available, to join in a chat or chat room, over a period of time, until we can be absolutely sure that everything is correct. One of these people should have a high speed connection because I don't any more.

I have a methodology to complete this task but I lack time and motivation to do it all on my own. I have a set that's 80% complete as it stands, but need help verifying and completing it. When done it would become the new torrent that I've been putting off for over two years, and could also be reuploaded to MediaFire.

[woogerboy21]

Nothing taken personally I assure you. Our sets do need a refresh. I just can't find anyone to work with to make it happen. I need someone(s) to be available, to join in a chat or chat room, over a period of time, until we can be absolutely sure that everything is correct. One of these people should have a high speed connection because I don't any more.

I have a methodology to complete this task but I lack time and motivation to do it all on my own. I have a set that's 80% complete as it stands, but need help verifying and completing it. When done it would become the new torrent that I've been putting off for over two years, and could also be reuploaded to MediaFire.

I would be willing to help out. But first I was curious about when you mentioned high speed internet. I have what is considered high speed in my area but what aspect were you looking for? If its upstream they are very restrictive on that side of the pipe. Other than that I wouldn't mind helping out on refreshing things.

[Ark]

Nothing taken personally I assure you. Our sets do need a refresh. I just can't find anyone to work with to make it happen. I need someone(s) to be available, to join in a chat or chat room, over a period of time, until we can be absolutely sure that everything is correct. One of these people should have a high speed connection because I don't any more.

I have a methodology to complete this task but I lack time and motivation to do it all on my own. I have a set that's 80% complete as it stands, but need help verifying and completing it. When done it would become the new torrent that I've been putting off for over two years, and could also be reuploaded to MediaFire.

I mentioned before that I have pretty much all the HQ pics unzipped and in folders on my ark42.com website like http://ark42.com/mtg/cards/Scars%20of%2 ... r.full.jpg, so that I can use them for things like http://ark42.com/mtg/search.php?name=Koth&pics=3
I try to keep up with all set releases and promos, and also fix typos so that the card names work with the actual oracle names, which means fixing uppercase/lowercase sometimes even.
If there is anything I can do to help, from a Linux command line with no GUI, I could. I have a 10Mbit upload and download connection on that server, but no way to run any Windows software there.

[woogerboy21]

I mentioned before that I have pretty much all the HQ pics unzipped and in folders on my ark42.com website like http://ark42.com/mtg/cards/Scars%20of%2 ... r.full.jpg, so that I can use them for things like http://ark42.com/mtg/search.php?name=Koth&pics=3
I try to keep up with all set releases and promos, and also fix typos so that the card names work with the actual oracle names, which means fixing uppercase/lowercase sometimes even.
If there is anything I can do to help, from a Linux command line with no GUI, I could. I have a 10Mbit upload and download connection on that server, but no way to run any Windows software there.

Your not wanting people to link to your site are you? That would probably eat up that bandwidth pretty quick if users begin snagging pictures from it. Also from the structure of images you have in place do you include extra's such as all tokens or tips and tricks. Also how about cropped forms of the images?

[Huggybaby]

Three is better than two for this.

Here's basically what we'll be doing:
First you must Torrentzip each set. This works on non windows machines too. Then, make an md5 checksum for each set using md5summer. Some of our sets won't match since I might have an out of date edition, or there may be a typo, etc.
For the sets that don't match, we unzip and fix the problem.

When all our sets match, I make a torrent. The sets in the torrent should be uploaded to a hoster as well, and Ark's 10Mb connection would come in real handy there.

[Ark]

Your not wanting people to link to your site are you? That would probably eat up that bandwidth pretty quick if users begin snagging pictures from it. Also from the structure of images you have in place do you include extra's such as all tokens or tips and tricks. Also how about cropped forms of the images?

No, I don't want 100s of people linking to my site for pictures. I'll put access controls in the .htaccess file to require valid referer headers and/or limit bandwidth if it comes to it, but I'd rather not as long as it's left to normal usage for the most part.

Also, no I do not have tips & tricks or tokens or crops. Only fulls. I really don't understand the point of crops.

[Ark]

Here's basically what we'll be doing:
First you must Torrentzip each set. This works on non windows machines too. Then, make an md5 checksum for each set using md5summer. Some of our sets won't match since I might have an out of date edition, or there may be a typo, etc.
For the sets that don't match, we unzip and fix the problem.

I can zip and md5sum with ease, but as far as uploading from the command line, I can't use any sort of web browser on the server other than Lynx, which probably can't even deal with basic javascript and cookies to log into websites to upload things.

[chaudakh]

For those interested in, I have already programed a solution designed to extract data from the Gatherer : http://www.mediafire.com/?3akc6mtapnar2
Program in VB.NET. Accordingly, Microsoft Framework .NET is required.

The program is in French by default, but I can easily make it multilingual if required. I don't know if it helps, but I think it basically does the same things as the different programs exhibited before.

[Ark]

For those interested in, I have already programed a solution designed to extract data from the Gatherer : http://www.mediafire.com/?3akc6mtapnar2

The program is in French by default, but I can easily make it multilingual if required. I don't know if it helps, but I think it basically does the same things as the different programs exhibited before.

Wow, this looks very similar to viewtopic.php?f=15&t=3371
I can't tell what it does different than mine, since I can't read French though.

[frwololo]

Hi woogerboy21 and thanks a lot for this application (and the wagic support!)
I'm running on Vista (a Japanese version...), and I can't seem to be able to start the application correctly.
I first get the error, I click on "no", and then the application is always in "initializing" state, never gets into "ready" mode. Is there something special that needs to be done at this point?

[Ark]

Hi woogerboy21 and thanks a lot for this application (and the wagic support!)
I'm running on Vista (a Japanese version...), and I can't seem to be able to start the application correctly.
I first get the error, I click on "no", and then the application is always in "initializing" state, never gets into "ready" mode. Is there something special that needs to be done at this point?

Hey if you want to try out mine (viewtopic.php?f=15&t=3371) let me know the format for "wagic" and I can easily add other outputs for the spoiler/card database.

[woogerboy21]

Hi woogerboy21 and thanks a lot for this application (and the wagic support!)
I'm running on Vista (a Japanese version...), and I can't seem to be able to start the application correctly.
I first get the error, I click on "no", and then the application is always in "initializing" state, never gets into "ready" mode. Is there something special that needs to be done at this point?

When you say you click "no" to the error, what does the error or popup actually say?

Ark -
I don't recall off the top of my head the wagic card name format (i think its by number) but I do remember that the application has its own folder structure format as well as once all the images are downloaded you can place the images in an uncompressed .zip file. Let me check the code again for the specifics (unless someone else knows off the top of there head)

edit:
Wagic names the cards by there multiverseid's (not card number). The images can be stored either in the sets abbreviation folder or zipped into a noncompressed zip file named with the sets abbreviation. (Ex: M10.zip)
Also Ark, I haven't used your application much but does it actually create the folder structure needed for various applications? Your app is great and I would consider it far superior than the hta I wrote but if it doesnt create the directory structure I believe people shy away.

[Ark]

I don't recall off the top of my head the wagic card name format (i think its by number) but I do remember that the application has its own folder structure format as well as once all the images are downloaded you can place the images in an uncompressed .zip file. Let me check the code again for the specifics (unless someone else knows off the top of there head)

edit:
Wagic names the cards by there multiverseid's (not card number). The images can be stored either in the sets abbreviation folder or zipped into a noncompressed zip file named with the sets abbreviation. (Ex: M10.zip)
Also Ark, I haven't used your application much but does it actually create the folder structure needed for various applications? Your app is great and I would consider it far superior than the hta I wrote but if it doesnt create the directory structure I believe people shy away.

The options are to create directories for the images per-set and you can use the set code like M10 or the full set name for the folder "Magic 2010". The files are created with the card name. There is a 3rd option which creates the files with the multiverse ID for the file name, but does not use any per-set folders. That is easy to change I guess, if people need an option for per-set folders with multiverse ID filenames though.

I guess what I thought was more important is the card database. Does Wagic need a specific file format for the card database, or does it already have the card database on its own?