Collectible Card Game Headquarters

Forgot to commit. It's up now.

Time to destroy your server :-d

Actually, not really... I left it with the 2.5 second delay. (2442 ms)

Rob Cashwalker wrote:Actually, not really... I left it with the 2.5 second delay. (2442 ms)

11000+ cards * 2.5 sec / 60 (sec/min) = 458+ min. I'm too impatient for that. I'll just tweak the delay down locally.

Don't go crazy yet... I still have 7000 to grab myself, and I found that if it went too quickly, it stalled, so I'm letting it go naturally. The thing is, while it's going I'm finding out which recent cards need their pic counts fixed or if there are missing or miss-named pics in my archive.

I notice someone's downloading urls with "=" right now... 4 files for each 1 of mine.

More pics downloaded!

I started the day with around 850 missing and am now down to 651. From the console messages I am seeing when requesting a new download it appears that whenever there are two or more images for a specific card in a set that the second and subsequent images are not downloading properly.

Code: Select all

LRW/Swamp4.full.jpg http://cardforge.org/fpics/LRW/U3dhbXA0LmZ1bGwuanBn
LRW/Swamp4.full.jpg http://cardforge.org/fpics/LRW/U3dhbXA0LmZ1bGwuanBn
LRW/Swamp4.full.jpg http://cardforge.org/fpics/LRW/U3dhbXA0LmZ1bGwuanBn
2ED/Swamp3.full.jpg http://cardforge.org/fpics/U/U3dhbXAzLmZ1bGwuanBn
2ED/Swamp3.full.jpg http://cardforge.org/fpics/U/U3dhbXAzLmZ1bGwuanBn

My LRW folder contains Swamp4.full.jpg and no other swamps.
MY 2ED folder contains Swamp3.full,jpg and no other swamps.

Before I started downloading the LQ by sets a couple weeks ago I removed all my jpgs from the pics folder, created the set folders and started with a clean folder structure.

I am now down to only 19 missing.

Jaedayr - It's likely the PicCount for those sets may be wrong.

Chris - Post the console trace with the download errors, so I can see which pics you had trouble with.

Rob Cashwalker wrote:Chris - Post the console trace with the download errors, so I can see which pics you had trouble with.

`
Will do, I just updated my local copy to rev7365 and was then able to download most of the last 19 missing pictures.

I then tried to download the now remaining 14 missing pics and this is the console listing:

Code: Select all

CHR/Urza's Power Plant4.full.jpg http://cardforge.org/fpics/CH/VXJ6YSdzIFBvd2VyIFBsYW50NC5mdWxsLmpwZw
CHR/Urza's Power Plant4.full.jpg http://cardforge.org/fpics/CH/VXJ6YSdzIFBvd2VyIFBsYW50NC5mdWxsLmpwZw
CHR/Urza's Power Plant4.full.jpg http://cardforge.org/fpics/CH/VXJ6YSdzIFBvd2VyIFBsYW50NC5mdWxsLmpwZw
CHR/Urza's Tower4.full.jpg http://cardforge.org/fpics/CH/VXJ6YSdzIFRvd2VyNC5mdWxsLmpwZw
CHR/Urza's Tower4.full.jpg http://cardforge.org/fpics/CH/VXJ6YSdzIFRvd2VyNC5mdWxsLmpwZw
CHR/Urza's Tower4.full.jpg http://cardforge.org/fpics/CH/VXJ6YSdzIFRvd2VyNC5mdWxsLmpwZw
CHR/Urza's Mine4.full.jpg http://cardforge.org/fpics/CH/VXJ6YSdzIE1pbmU0LmZ1bGwuanBn
CHR/Urza's Mine4.full.jpg http://cardforge.org/fpics/CH/VXJ6YSdzIE1pbmU0LmZ1bGwuanBn
CHR/Urza's Mine4.full.jpg http://cardforge.org/fpics/CH/VXJ6YSdzIE1pbmU0LmZ1bGwuanBn
FEM/Homarid Warrior.full.jpg http://cardforge.org/fpics/FE/SG9tYXJpZCBXYXJyaW9yLmZ1bGwuanBn
FEM/Dwarven Soldier.full.jpg http://cardforge.org/fpics/FE/RHdhcnZlbiBTb2xkaWVyLmZ1bGwuanBn
FEM/Orcish Spy.full.jpg http://cardforge.org/fpics/FE/T3JjaXNoIFNweS5mdWxsLmpwZw
ALL/Gift of the Woods.full.jpg http://cardforge.org/fpics/AL/R2lmdCBvZiB0aGUgV29vZHMuZnVsbC5qcGc
FEM/Elvish Scout.full.jpg http://cardforge.org/fpics/FE/RWx2aXNoIFNjb3V0LmZ1bGwuanBn
basedir: res/pics
0/14
1/14
2/14
3/14
4/14
5/14
6/14
7/14
8/14
9/14
Error - the LQ picture for FEM/Homarid Warrior.full.jpg could not be found on the server. [http://cardforge.org/fpics/FE/SG9tYXJpZCBXYXJyaW9yLmZ1bGwuanBn] - http://cardforge.org/fpics/FE/SG9tYXJpZCBXYXJyaW9yLmZ1bGwuanBn
10/14
Error - the LQ picture for FEM/Dwarven Soldier.full.jpg could not be found on the server. [http://cardforge.org/fpics/FE/RHdhcnZlbiBTb2xkaWVyLmZ1bGwuanBn] - http://cardforge.org/fpics/FE/RHdhcnZlbiBTb2xkaWVyLmZ1bGwuanBn
11/14
Error - the LQ picture for FEM/Orcish Spy.full.jpg could not be found on the server. [http://cardforge.org/fpics/FE/T3JjaXNoIFNweS5mdWxsLmpwZw] - http://cardforge.org/fpics/FE/T3JjaXNoIFNweS5mdWxsLmpwZw
12/14
Error - the LQ picture for ALL/Gift of the Woods.full.jpg could not be found on the server. [http://cardforge.org/fpics/AL/R2lmdCBvZiB0aGUgV29vZHMuZnVsbC5qcGc] - http://cardforge.org/fpics/AL/R2lmdCBvZiB0aGUgV29vZHMuZnVsbC5qcGc
13/14
Error - the LQ picture for FEM/Elvish Scout.full.jpg could not be found on the server. [http://cardforge.org/fpics/FE/RWx2aXNoIFNjb3V0LmZ1bGwuanBn] - http://cardforge.org/fpics/FE/RWx2aXNoIFNjb3V0LmZ1bGwuanBn
14/14

`
I looks like some of these 14 pics are being downloaded but when I go back to try again it states that there is still 14 pics to download. Nonetheless, I think that we are now getting very close to getting this to work.

That's interesting that it's trying to download the Chronicles versions, since we don't use Chronicles at all. Maybe that's part of the problem for a few of those? I'm pretty sure the other 5 all have multiple pictures in that set.

Sorry for appearing so clueless but I really do not know what is going on or how to fix it. My swamp.txt is r6026 2/1/2011 by Rob. It contains the following.

Code: Select all

Name:Swamp
ManaCost:no cost
Types:Basic Land Swamp
Text:no text
A:AB$ Mana | Cost$ T | Produced$ B | SpellDescription$ Add B to your mana pool.
SVar:PicCount:4
SVar:Rarity:Land
SVar:Picture:http://resources.wizards.com/magic/cards/unh/en-us/card73973.jpg\http://gatherer.wizards.com/handlers/image.ashx?type=card&multiverseid=159290\http://gatherer.wizards.com/handlers/image.ashx?type=card&multiverseid=159291\http://gatherer.wizards.com/handlers/image.ashx?type=card&multiverseid=159292
SetInfo:USG|Land|http://magiccards.info/scans/en/us/339.jpg|4
SetInfo:TMP|Land|http://magiccards.info/scans/en/tp/334.jpg|4
SetInfo:ZEN|Land|http://magiccards.info/scans/en/zen/238.jpg|4
SetInfo:4ED|Land|http://magiccards.info/scans/en/4e/190.jpg|3
SetInfo:3ED|Land|http://magiccards.info/scans/en/rv/299.jpg|3
SetInfo:SOM|Land|http://magiccards.info/scans/en/som/238.jpg|4
SetInfo:TSP|Land|http://magiccards.info/scans/en/ts/290.jpg|4
SetInfo:ODY|Land|http://magiccards.info/scans/en/od/339.jpg|4
SetInfo:MIR|Land|http://magiccards.info/scans/en/mr/311.jpg|4
SetInfo:PTK|Land|http://magiccards.info/scans/en/p3k/172.jpg|3
SetInfo:7ED|Land|http://magiccards.info/scans/en/7e/346.jpg|4
SetInfo:POR|Land|http://magiccards.info/scans/en/po/219.jpg|4
SetInfo:ROE|Land|http://magiccards.info/scans/en/roe/237.jpg|4
SetInfo:10E|Land|http://magiccards.info/scans/en/10e/372.jpg|4
SetInfo:RAV|Land|http://magiccards.info/scans/en/rav/295.jpg|4
SetInfo:6ED|Land|http://magiccards.info/scans/en/6e/339.jpg|4
SetInfo:5ED|Land|http://magiccards.info/scans/en/5e/442.jpg|4
SetInfo:ONS|Land|http://magiccards.info/scans/en/on/339.jpg|4
SetInfo:S99|Land|http://magiccards.info/scans/en/st/162.jpg|4
SetInfo:MBS|Land|http://magiccards.info/scans/en/mbs/150.jpg|2
SetInfo:8ED|Land|http://magiccards.info/scans/en/8e/339.jpg|4
SetInfo:CHK|Land|http://magiccards.info/scans/en/chk/295.jpg|4
SetInfo:PO2|Land|http://magiccards.info/scans/en/po2/163.jpg|3
SetInfo:9ED|Land|http://magiccards.info/scans/en/9e/339.jpg|4
SetInfo:ICE|Land|http://magiccards.info/scans/en/ia/353.jpg|3
SetInfo:MRD|Land|http://magiccards.info/scans/en/mi/295.jpg|4
SetInfo:SHM|Land|http://magiccards.info/scans/en/shm/290.jpg|4
SetInfo:LEA|Land|http://magiccards.info/scans/en/al/290.jpg|2
SetInfo:LEB|Land|http://magiccards.info/scans/en/be/295.jpg|3
SetInfo:INV|Land|http://magiccards.info/scans/en/in/339.jpg|4
SetInfo:MMQ|Land|http://magiccards.info/scans/en/mm/339.jpg|4
SetInfo:ALA|Land|http://magiccards.info/scans/en/ala/238.jpg|4
SetInfo:M11|Land|http://magiccards.info/scans/en/m11/238.jpg|4
SetInfo:M10|Land|http://magiccards.info/scans/en/m10/238.jpg|4
SetInfo:LRW|Land|http://magiccards.info/scans/en/lw/290.jpg|4
SetInfo:2ED|Land|http://magiccards.info/scans/en/un/295.jpg|3
End

2ED in the file above says there are 3 pics for swamp. If I look at Gatherer that also seems to indicate 3 pics for swamp in Unlimited/2ED. But as noted in a previous post, I have only one swamp in my 2ED folder and have had only one for the last two weeks of running almost daily updates and syncs. What am I doing wrong or not doing?

OK, I fixed a bug in building the filenames... this explains why pictures {1 to PicCount-1} were being re-downloaded.

I also added estimated time remaining to the progress bar and randomized the delay from .5 seconds to 1/5 seconds.

Earlier I patched a number of the FEM and ALL cards to include PicCount.

YES!!!!!!!!

I now have downloaded EVERY SET PIC.

I agree, it is great to finally have all of the pics. Good job Rob. Thank you.