Gatherer updated: Crawler doesn't work anymore

[LevelX]

The WOTC Gatherer card database site was updated recently.

North are you able to check and update if possible the Gatherer Crawler to work with the new Gatherer?

[North]

Thanks for letting me know. I'll try to get it done as soon as possible.

[North]

They only changed the CSS (styling of the pages). This change has no impact on the crawler itself. I did test it and it worked just fine.

[LevelX]

Maybe it's a edition specific problem.

At least if I set only one expansion with

Code: Select all

sets.add("Magic: The Gathering—Conspiracy");

I only get one card of Conspiracy in the output files.

If I go for all sets I get the following error message after a while:

Code: Select all

http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Alara+Reborn"]|["Alliances"]|["Antiquities"]|["Apocalypse"]|["Arabian+Nights"]|["Archenemy"]|["Avacyn+Restored"]|["Battle+Royale+Box+Set"]|["Beatdown+Box+Set"]|["Betrayers+of+Kamigawa"]|["Born+of+the+Gods"]|["Champions+of+Kamigawa"]|["Chronicles"]|["Classic+Sixth+Edition"]|["Coldsnap"]|["Commander+2013+Edition"]|["Commander's+Arsenal"]|["Conflux"]|["Dark+Ascension"]|["Darksteel"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Dissension"]|["Dragon's+Maze"]|["Duel+Decks:+Ajani+vs.+Nicol+Bolas"]|["Duel+Decks:+Divine+vs.+Demonic"]|["Duel+Decks:+Elspeth+vs.+Tezzeret"]|["Duel+Decks:+Elves+vs.+Goblins"]|["Duel+Decks:+Garruk+vs.+Liliana"]|["Duel+Decks:+Heroes+vs.+Monsters"]|["Duel+Decks:+Izzet+vs.+Golgari"]|["Duel+Decks:+Jace+vs.+Chandra"]|["Duel+Decks:+Jace+vs.+Vraska"]|["Duel+Decks:+Knights+vs.+Dragons"]|["Duel+Decks:+Phyrexia+vs.+the+Coalition"]|["Duel+Decks:+Sorin+vs.+Tibalt"]|["Duel+Decks:+Venser+vs.+Koth"]|["Eighth+Edition"]|["Eventide"]|["Exodus"]|["Fallen+Empires"]|["Fifth+Dawn"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Fifth+Edition"]|["Fourth+Edition"]|["From+the+Vault:+Dragons"]|["From+the+Vault:+Exiled"]|["From+the+Vault:+Legends"]|["From+the+Vault:+Realms"]|["From+the+Vault:+Relics"]|["From+the+Vault:+Twenty"]|["Future+Sight"]|["Gatecrash"]|["Guildpact"]|["Homelands"]|["Ice+Age"]|["Innistrad"]|["Invasion"]|["Journey+into+Nyx"]|["Judgment"]|["Legends"]|["Legions"]|["Limited+Edition+Alpha"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Limited+Edition+Beta"]|["Lorwyn"]|["Magic+2010"]|["Magic+2011"]|["Magic+2012"]|["Magic+2013"]|["Magic+2014+Core+Set"]|["Magic:+The+Gathering-Commander"]|["Magic:+The+Gathering—Conspiracy"]|["Masters+Edition"]|["Masters+Edition+II"]|["Masters+Edition+III"]|["Masters+Edition+IV"]|["Mercadian+Masques"]|["Mirage"]|["Mirrodin"]|["Mirrodin+Besieged"]|["Modern+Event+Deck+2014"]|["Modern+Masters"]|["Morningtide"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Nemesis"]|["New+Phyrexia"]|["Ninth+Edition"]|["Odyssey"]|["Onslaught"]|["Planar+Chaos"]|["Planechase"]|["Planechase+2012+Edition"]|["Planeshift"]|["Portal"]|["Portal+Second+Age"]|["Portal+Three+Kingdoms"]|["Premium+Deck+Series:+Fire+and+Lightning"]|["Premium+Deck+Series:+Graveborn"]|["Premium+Deck+Series:+Slivers"]|["Promo+set+for+Gatherer"]|["Prophecy"]|["Ravnica:+City+of+Guilds"]|["Return+to+Ravnica"]|["Revised+Edition"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Rise+of+the+Eldrazi"]|["Saviors+of+Kamigawa"]|["Scars+of+Mirrodin"]|["Scourge"]|["Seventh+Edition"]|["Shadowmoor"]|["Shards+of+Alara"]|["Starter+1999"]|["Starter+2000"]|["Stronghold"]|["Tempest"]|["Tenth+Edition"]|["The+Dark"]|["Theros"]|["Time+Spiral"]|["Time+Spiral+"Timeshifted""]|["Torment"]|["Unlimited+Edition"]|["Urza's+Destiny"]|["Urza's+Legacy"]
http://gatherer.wizards.com/Pages/Search/Default.aspx?action=advanced&output=checklist&set=|["Urza's+Saga"]|["Vanguard"]|["Vintage+Masters"]|["Visions"]|["Weatherlight"]|["Worldwake"]|["Zendikar"]
Exception in thread "Thread-1" java.util.NoSuchElementException
   at java.util.AbstractQueue.remove(AbstractQueue.java:117)
   at north.gatherercrawler.util.ParseQueue.remove(ParseQueue.java:23)
   at north.gatherercrawler.CardParser.run(CardParser.java:237)

[North]

Last week I ran this over all the sets and it seemed to work.

On Friday I was having trouble getting the Conspiracy cards too but when I tried to fix it today it worked just fine. Who knows what the problem was. If it does misbehave again please let me know.

[LevelX]

Sorry, but I still have problems.

Other sets than conspiracy seem to work. But Conspiracy does not.
I suppose it has to do with the set name of consipracy in the Gatherer.
Magic: The Gathering-Conspiracy
It has a long dash in the name.
If I execute the created url from the browser it works. The list of conspiracy cards is shown.
But if I debug Gatherer Crawler and show the returned doc given back with the same url, the document contains no card information.
I guess the long dash char gets somehow wrong by character set conversion.


EDIT: I got it to work by using
sets.add("Magic:+The+Gathering%E2%80%94Conspiracy");
as set name.

EDIT2:
But getting only 197 of 210 cards. The 13 missing cards are the cards with the new card type "Conspiracy". But I'm not sure if this is a bug of Gatherer or the Gatherer Crawler. The cards are available in Gatherer by name search. However if I search in Gatherer with the card type = "Conspiracy", I get an empty list.

PS:
Is there something to change/add in the card tracker to add "Conspiracy" cards?

[North]

I took a look and there seems to be no limitation on what type a card can be. You can go ahead and add them as normal.