Gatherer updated: Crawler doesn't work anymore
Moderators: North, BetaSteward, noxx, jeffwadsworth, JayDi, TheElk801, LevelX, CCGHQ Admins
Gatherer updated: Crawler doesn't work anymore
by LevelX » 15 Jun 2014, 13:10
The WOTC Gatherer card database site was updated recently.
North are you able to check and update if possible the Gatherer Crawler to work with the new Gatherer?
North are you able to check and update if possible the Gatherer Crawler to work with the new Gatherer?
-
LevelX - DEVELOPER
- Posts: 1677
- Joined: 08 Dec 2011, 15:08
- Has thanked: 174 times
- Been thanked: 374 times
Re: Gatherer updated: Crawler doesn't work anymore
by North » 16 Jun 2014, 18:44
Thanks for letting me know. I'll try to get it done as soon as possible.
Re: Gatherer updated: Crawler doesn't work anymore
by North » 17 Jun 2014, 07:50
They only changed the CSS (styling of the pages). This change has no impact on the crawler itself. I did test it and it worked just fine.
Re: Gatherer updated: Crawler doesn't work anymore
by LevelX » 19 Jun 2014, 10:17
Maybe it's a edition specific problem.
At least if I set only one expansion with
If I go for all sets I get the following error message after a while:
At least if I set only one expansion with
- Code: Select all
sets.add("Magic: The Gathering—Conspiracy");
If I go for all sets I get the following error message after a while:
- Code: Select all
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Alara+Reborn"]|["Alliances"]|["Antiquities"]|["Apocalypse"]|["Arabian+Nights"]|["Archenemy"]|["Avacyn+Restored"]|["Battle+Royale+Box+Set"]|["Beatdown+Box+Set"]|["Betrayers+of+Kamigawa"]|["Born+of+the+Gods"]|["Champions+of+Kamigawa"]|["Chronicles"]|["Classic+Sixth+Edition"]|["Coldsnap"]|["Commander+2013+Edition"]|["Commander's+Arsenal"]|["Conflux"]|["Dark+Ascension"]|["Darksteel"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Dissension"]|["Dragon's+Maze"]|["Duel+Decks:+Ajani+vs.+Nicol+Bolas"]|["Duel+Decks:+Divine+vs.+Demonic"]|["Duel+Decks:+Elspeth+vs.+Tezzeret"]|["Duel+Decks:+Elves+vs.+Goblins"]|["Duel+Decks:+Garruk+vs.+Liliana"]|["Duel+Decks:+Heroes+vs.+Monsters"]|["Duel+Decks:+Izzet+vs.+Golgari"]|["Duel+Decks:+Jace+vs.+Chandra"]|["Duel+Decks:+Jace+vs.+Vraska"]|["Duel+Decks:+Knights+vs.+Dragons"]|["Duel+Decks:+Phyrexia+vs.+the+Coalition"]|["Duel+Decks:+Sorin+vs.+Tibalt"]|["Duel+Decks:+Venser+vs.+Koth"]|["Eighth+Edition"]|["Eventide"]|["Exodus"]|["Fallen+Empires"]|["Fifth+Dawn"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Fifth+Edition"]|["Fourth+Edition"]|["From+the+Vault:+Dragons"]|["From+the+Vault:+Exiled"]|["From+the+Vault:+Legends"]|["From+the+Vault:+Realms"]|["From+the+Vault:+Relics"]|["From+the+Vault:+Twenty"]|["Future+Sight"]|["Gatecrash"]|["Guildpact"]|["Homelands"]|["Ice+Age"]|["Innistrad"]|["Invasion"]|["Journey+into+Nyx"]|["Judgment"]|["Legends"]|["Legions"]|["Limited+Edition+Alpha"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Limited+Edition+Beta"]|["Lorwyn"]|["Magic+2010"]|["Magic+2011"]|["Magic+2012"]|["Magic+2013"]|["Magic+2014+Core+Set"]|["Magic:+The+Gathering-Commander"]|["Magic:+The+Gathering—Conspiracy"]|["Masters+Edition"]|["Masters+Edition+II"]|["Masters+Edition+III"]|["Masters+Edition+IV"]|["Mercadian+Masques"]|["Mirage"]|["Mirrodin"]|["Mirrodin+Besieged"]|["Modern+Event+Deck+2014"]|["Modern+Masters"]|["Morningtide"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Nemesis"]|["New+Phyrexia"]|["Ninth+Edition"]|["Odyssey"]|["Onslaught"]|["Planar+Chaos"]|["Planechase"]|["Planechase+2012+Edition"]|["Planeshift"]|["Portal"]|["Portal+Second+Age"]|["Portal+Three+Kingdoms"]|["Premium+Deck+Series:+Fire+and+Lightning"]|["Premium+Deck+Series:+Graveborn"]|["Premium+Deck+Series:+Slivers"]|["Promo+set+for+Gatherer"]|["Prophecy"]|["Ravnica:+City+of+Guilds"]|["Return+to+Ravnica"]|["Revised+Edition"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Rise+of+the+Eldrazi"]|["Saviors+of+Kamigawa"]|["Scars+of+Mirrodin"]|["Scourge"]|["Seventh+Edition"]|["Shadowmoor"]|["Shards+of+Alara"]|["Starter+1999"]|["Starter+2000"]|["Stronghold"]|["Tempest"]|["Tenth+Edition"]|["The+Dark"]|["Theros"]|["Time+Spiral"]|["Time+Spiral+"Timeshifted""]|["Torment"]|["Unlimited+Edition"]|["Urza's+Destiny"]|["Urza's+Legacy"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Urza's+Saga"]|["Vanguard"]|["Vintage+Masters"]|["Visions"]|["Weatherlight"]|["Worldwake"]|["Zendikar"]
Exception in thread "Thread-1" java.util.NoSuchElementException
at java.util.AbstractQueue.remove(AbstractQueue.java:117)
at north.gatherercrawler.util.ParseQueue.remove(ParseQueue.java:23)
at north.gatherercrawler.CardParser.run(CardParser.java:237)
-
LevelX - DEVELOPER
- Posts: 1677
- Joined: 08 Dec 2011, 15:08
- Has thanked: 174 times
- Been thanked: 374 times
Re: Gatherer updated: Crawler doesn't work anymore
by North » 23 Jun 2014, 08:09
Last week I ran this over all the sets and it seemed to work.
On Friday I was having trouble getting the Conspiracy cards too but when I tried to fix it today it worked just fine. Who knows what the problem was. If it does misbehave again please let me know.
On Friday I was having trouble getting the Conspiracy cards too but when I tried to fix it today it worked just fine. Who knows what the problem was. If it does misbehave again please let me know.
Re: Gatherer updated: Crawler doesn't work anymore
by LevelX » 23 Jun 2014, 11:35
Sorry, but I still have problems.
Other sets than conspiracy seem to work. But Conspiracy does not.
I suppose it has to do with the set name of consipracy in the Gatherer.
Magic: The Gathering-Conspiracy
It has a long dash in the name.
If I execute the created url from the browser it works. The list of conspiracy cards is shown.
But if I debug Gatherer Crawler and show the returned doc given back with the same url, the document contains no card information.
I guess the long dash char gets somehow wrong by character set conversion.
EDIT: I got it to work by using
sets.add("Magic:+The+Gathering%E2%80%94Conspiracy");
as set name.
EDIT2:
But getting only 197 of 210 cards. The 13 missing cards are the cards with the new card type "Conspiracy". But I'm not sure if this is a bug of Gatherer or the Gatherer Crawler. The cards are available in Gatherer by name search. However if I search in Gatherer with the card type = "Conspiracy", I get an empty list.
PS:
Is there something to change/add in the card tracker to add "Conspiracy" cards?
Other sets than conspiracy seem to work. But Conspiracy does not.
I suppose it has to do with the set name of consipracy in the Gatherer.
Magic: The Gathering-Conspiracy
It has a long dash in the name.
If I execute the created url from the browser it works. The list of conspiracy cards is shown.
But if I debug Gatherer Crawler and show the returned doc given back with the same url, the document contains no card information.
I guess the long dash char gets somehow wrong by character set conversion.
EDIT: I got it to work by using
sets.add("Magic:+The+Gathering%E2%80%94Conspiracy");
as set name.
EDIT2:
But getting only 197 of 210 cards. The 13 missing cards are the cards with the new card type "Conspiracy". But I'm not sure if this is a bug of Gatherer or the Gatherer Crawler. The cards are available in Gatherer by name search. However if I search in Gatherer with the card type = "Conspiracy", I get an empty list.
PS:
Is there something to change/add in the card tracker to add "Conspiracy" cards?
-
LevelX - DEVELOPER
- Posts: 1677
- Joined: 08 Dec 2011, 15:08
- Has thanked: 174 times
- Been thanked: 374 times
Re: Gatherer updated: Crawler doesn't work anymore
by North » 23 Jun 2014, 15:34
I took a look and there seems to be no limitation on what type a card can be. You can go ahead and add them as normal.
7 posts
• Page 1 of 1
Who is online
Users browsing this forum: No registered users and 12 guests