It is currently 14 Dec 2017, 04:12
   
Text Size

LordHelmchen's price import - BFZ and mkm

Moderators: charmer, CCGHQ Admins

Re: LordHelmchen's price import (lua) (updated 2014-10-28)

Postby Golob » 13 Jul 2015, 05:37

Hello,
is there any chance for an update to the LHpi Data? The script is way too good for not being updated. :D
Golob
 
Posts: 16
Joined: 06 Aug 2010, 13:06
Has thanked: 0 time
Been thanked: 1 time

state of things

Postby LordHelmchen » 13 Jul 2015, 13:02

Golob wrote:Hello,
is there any chance for an update to the LHpi Data? The script is way too good for not being updated. :D
There is.
With that said, it may not be as soon as hoped. As a player, I am no longer interested in prices besides those at http://magickartenmarkt.de/ (magiccardmarket.eu).
The Library, Data and helper scripts will be updated according to LHpi.magickartenmarkt.lua's needs and new MtG set's releases.
After http://www.slightlymagic.net/forum/viewtopic.php?f=32&t=15942&view=viewpoll suggests that hardly anybody actually needed the scripts, sitescripts besides the new LHpi.magickartenmarkt.lua (and probably LHpi.tcgplayerPriceGuide.lua) will only be updated if I either need them to sanity check the library or if someone specifically requests it.
As LHpi is gpl'ed open source and the documentation is a wiki, feel free to contribute. If there is a raised interest for it, I could look for a git hoster and upload my development history (and keep it in sync with further developments)

Releases of new versions will be made once I have reached a state that adheres to my self imposed usability and quality standards.
I am sorry for the inconveniances this may (have) cause(d) you.

To keep all relevant info in one place, let me quote myself from http://www.slightlymagic.net/forum/viewtopic.php?f=32&t=17250#p179785:
LordHelmchen on 28 May 2015 in another thread wrote:I have been trying to do that for a long time with my LHpi library.
The first try (around LHpi-v2.8 ) by html-scraping turned out not to work, as lots of crucial information was not included when requesting a whole set. Despite wanting to do MKM prices, I turned to lower hanging fruit.
I returned to try importing from MKM October last year, and went the mkm-api route. That aproach was promising, but despite reaching a semi-working alpha I hit some stopgaps and eventually temporally lost my motivation. see tracker.php?p=2&t=488 for details. To save you from trying: that version will not work today.
A month ago I resumed my work, opting to defer downloading data from the api to a seperate script, that has to be run outside of MA as the currently best compromise. Turns out, due to server strain they removed pricing information from the whole-set-request, so instead of ~200 request the downloader has to do ~30.000 single-card-requests. This resulted in "429 Too Many Requests" http errors. I'm currently awaiting a reply from mkm tech support to decide if it's worth pursuing this route.
It would probably work well enough to, say, update only standard sets once a week, but before I feel ok with releasing it, there's still a lot of documentation, testing and error-poofing to be done.
EDIT:
It would probably be a good idea to indeed add all new sets and check the library for backwards compatibility, after all mkm-related changes that have been made. You may well see an official LHpi.tcgplayerPriceGuide.lua update soonish :-"
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

Re: LordHelmchen's price import (lua) (updated 2014-10-28)

Postby Golob » 13 Jul 2015, 18:09

If you specify in clear what to do, I will willingly contribute my labour :)
Golob
 
Posts: 16
Joined: 06 Aug 2010, 13:06
Has thanked: 0 time
Been thanked: 1 time

Re: LordHelmchen's price import (lua) (updated 2014-10-28)

Postby LordHelmchen » 13 Jul 2015, 18:25

Golob wrote:If you specify in clear what to do, I will willingly contribute my labour :)
Give me a few days after MA updates to include Origins, and I'll at least release LHpi.tcgplayerPriceGuide.lua, LHpi-v2.15 and LHpi.Data-v6 and LHpi.sitescriptTemplate.
Afterwards, if you need an update to one of the other sitescripts, please take a few minutes to compare the script in question with the template and the tcgplayer script, and take a look at http://www.slightlymagic.net/wiki/How-T ... ptTemplate. If afterwards there are any questions left, I'll be happy to answer them :-)
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

Origins Update

Postby LordHelmchen » 18 Jul 2015, 18:27

This is probably the day where price import is the most useful, so here's at least the release of the new (Origins enabled) version with working tcgplayerPriceGuide.
Documentation at http://www.slightlymagic.net/wiki/LHpi will be updated later, probably :mrgreen:

Changelog library | Open
2.14 (unreleased)
cardcount.nontr renamed to cardcount.nontrad
GetSourceData split into GetSourceData and ParseSourceData.
* this change is completely transparent to existing sitescripts, but having GetSourceData seperated allows to use it from the sitescript for special cases.
LHpi.GetSourceData( sourceurl,urldetails )
* defer fetching to sitescript for oauth
* now returns #string sourcedata
* converts url to filename if OFFLINE
* fixed savepath==nil handling (needed when called before DoImport has run)
* Log https status if not ok
LHpi.ParseSourceData( sourcedata,sourceurl,urldetails )
* does the parsing previously done by GetSourceData and returns the #table sourceTable
* shortcut return nil for sourcedata==nil
* now strictly expect site.ParseHtmlData to return card.price as #number (and not #table, as was possible before)
* do not sanitize nor check card.regprice or card.foilpriceBuildCardData
* counts namereplace and foiltweak even if card is dropped
BuildCardData
* remove "(nontrad)","(replica)","(plane)","(scheme)","(conspiracy)" and
"(oversized)" card.name infixes when no longer needed
* set objecttype replica for 105,106,69
* improved Token suffix handling
* keeps "Replica" suffix for variants
* removes "Insert" suffix
* keep (oversized) in set [40]
SetPrice
* objtype infixe handling improved
Logtable
* no longer strictly requires #table arguments
MainImportCycle
* fixed CHECKEXPECTED handling for semi-available languages
DoImport
* changed site.expected.EXPECTTOKENS to site.expected.tokens
* changed site.expected.EXPECTNONTRAD to site.expected.nontrad
* changed site.expected.EXPECTREPL to site.expected.replica
* all three can be #boolean (like before) or #table { [langid]=#boolean,... }
* DoImport passes supImportfoil,supImportlangs,supImportsets to site.SetExpected
* fixed expected defaults and checks
removed some leftover commented-out code
improved some comments and log msgs
2.15
Split Initialize() from DoImport(importfoil , importlangs , importsets)
always run Initialize() before returning LHpi library table.
scriptname,savepath,logfile change from global to local (in site and LHpi)
Log now filters out VERBOSE and DEBUG lines by loglevel param to save "if DEBUG then Log() end" conditionals
*those conditionals have been cut and loglevel has been set for all existing LHpi.Log calls
DEBUGVARIANTS=true no longer sets DEBUG=false, instead remembers and restores previous DEBUG state
new LHpi.OAuthEncode(s) moved here from mkm-helper
workdir fixes
no longer check for Data in deprecated location

Changelog datafile | Open
v6
new 52,53,55,814,815,816,817,818,819,820,821,822
added lang [17]="Phyrexian"
fixed names for 21,40,814
variant table for 15,22,25,26,27,33,40,45
foiltweak table for 22,26,33
cardcount fixed/updated in 9,105,106,762,791,800,813,816,30,22,21,26
fixed/updated variants for 43,69,201[ITA],758,764,766,781,787,800,813,22
commented variants in 180,140,110,100,90,280,230,220,170,260,105,25
foiltweak fixed/updated in 813

Changelog LHpi.tcgplayerPriceGuide | Open
2.13.5.14
added 814
2.14.5.14
removed url to filename changes that are done by the library if OFFLINE
2.15.6.14
added 816,818,815,817,819,820,821,40,32,27,25,23,22,21,822,340,55,45,822
new features from template/mkm branch
site.BuildUrl
* supports multiple urls per set
* set url.details.setid
special handling for
* Dragonfury cards
* unusual lang cards
updated/fixed/added variants,namereplacements,expected

Changelog template | Open
2.14.5.12
removed url to filename changes that are done by the library if OFFLINE
2.15.6.13
script configuration variables local instead of global
workdir support
site.LoadLib split from ImportPrice
site.Initialize loads LHpi lib if not yet available (needed if ImportPrice is not called)
keep LHpi library table if already present
no longer check for lib and Data in deprecated location
removed site.CompareSiteSets
optional site.BuildUrl("list") behaviour
new optional function site.FetchExpansionList
site.BuildUrl supports multiple urls per set

Changelog dummyMA | Open
0.6
new dummy.CompareDummySets(mapath,libver)
new dummy.CompareDataSets(mapath,libver,dataver)
new dummy.CompareSiteSets (moved from LHpi.magickartenmarkt.lua)
new dummy.ListUnknownUrls(expansions,missing,file) (moved from LHpi.mkm-helper.lua)
error handling in ma.GetFile and ma.Putfile
for libver > 2.14 and Lua 5.2, use dofile instead of homemade loader
CompareDataSets compares with dummy's set tales instead of Database/Sets.txt
use global workdir as introduced in 2.15
added 52,53,55,814,815,816,817,818,819,820,821
fixed dummy.forceEnv
fixed savepath handling
deprecated dummy.loadlibonly and dummy.loadscript
dummy.mergetables no longer changes param #table teins
update helper functions ouput to log instead od stdout
ma.GetUrl now uses luasockets instead of returning nil
Attachments
LHpi.tcgplayerPriceGuide.lua.zip
get the fixed version below instead
(27.64 KiB) Downloaded 86 times
LHpi-v2.15.lua.zip
(21 KiB) Downloaded 217 times
LHpi.Data-v6.lua.zip
(40.76 KiB) Downloaded 186 times
LHpi.sitescriptTemplate-v2.15.6.13.lua.zip
(13.08 KiB) Downloaded 140 times
dummyMA.zip
(11.03 KiB) Downloaded 146 times
Last edited by LordHelmchen on 19 Jul 2015, 10:38, edited 1 time in total.
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

Re: LordHelmchen's price import (lua) (Origins for tcgplayer

Postby Golob » 19 Jul 2015, 00:46

Thanks for the work.
When you run the script though, it crashes with some weird random ASCII
Image.

It crashed on random sets, because when you run it with core sets only, it will pass some 30% of times. Unfortunately, tt returns no prices even then.

Million thanks for putting so much effort into the script.
If you need any help just tell me.
Golob
 
Posts: 16
Joined: 06 Aug 2010, 13:06
Has thanked: 0 time
Been thanked: 1 time

Re: LordHelmchen's price import (lua) (Origins for tcgplayer

Postby Golob » 19 Jul 2015, 01:30

Ok, after some time spent looking into code I changed
Code: Select all
OFFLINE = true--download from dummy, only change to false for release
to
Code: Select all
OFFLINE = false--download from dummy, only change to false for release
and it started to work.
All the info is there! Double triple thanks for updating both the script and the data file. =D> =D> =D>

Also, on the plus side, I started to understand all logic behind your script. :D If you could just drop some hints on where how you collect data for replacement tables, I could help you with future releases.
Golob
 
Posts: 16
Joined: 06 Aug 2010, 13:06
Has thanked: 0 time
Been thanked: 1 time

Re: LordHelmchen's price import (lua) (Origins for tcgplayer

Postby LordHelmchen » 19 Jul 2015, 09:01

First things first:
LHpi.tcgplayerPriceGuide.lua.zip
fixed: OFFLINE to default (false)
(27.65 KiB) Downloaded 159 times


I guess the "!! GetFile failed for ..." lines in $PATH_TO_MA\Prices\LHpi.tcgplayerPriceGuide.log were a dead giveaway on what went wrong there...
The canonical way (and the only change the above hotfix file contains) is to comment out the line, similar to how all other options that are also left for the library to set to default values, but yours is no less valid :-P
nitpick | Open
actually, as you've kept the comment I made (specifically to remind myself to change it before I release it #-o ) "download from dummy" is not a correct description after "OFFLINE=false" :lol:


As for the error (and crash) itself: thanks for pointing out a reliable way to reproduce it. This seems to be a MA bug. The last lines of the script that get executed are
Code: Select all
   LHpi.Log( "Lua script " .. scriptname .. " finished" ,0)
   collectgarbage()--try prevent MA crashes on exit
   ma.Log( "Lua script " .. scriptname .. " finished" )
end -- function ImportPrice
and since both $PATH_TO_MA\Prices\LHpi.tcgplayerPriceGuide.log and $PATH_TO_MA\Magic Album.log show those lines, I can only conclude that my script has indeed finished at the point of the crash. Fortunately, the problem is easily avoided.

For now, I would love you to use your newly gained insight into the scripts to check http://www.slightlymagic.net/wiki/How-To_sitescriptTemplate for general understandability... Feel free to improve the page :)
I have not written an "updating sitescripts to new library/template version - 2.13 TO 2.15" section yet.

I will shortly update the other sitescripts (they already run with lib version 2.15.6 without errors) to include the update helper function site.FetchExpansionList(), and use it to update site.sets with all available sets. What I won't do this time is site.namereplace,site.foiltweak tables for the new sets and site.expected (which is read if CHECKEXPECTED is left at the default true and shown in the log if VERBOSE is set to true).

If you could just drop some hints on where how you collect data for replacement tables, I could help you with future releases.
You will probably need multiple runs of the script to determine everything, so to save time and bandwidth, I suggest the following preparation:
example with LHpi.tcgplayerPriceGuide.lua
  • create a new $PATH_TO_MA\Prices\LHpi.tcgplayerPriceGuide\ directory
  • edit LHpi.tcgplayerPriceGuide.lua to set SAVEHTML=true
  • run the script to download all source html files
  • edit LHpi.tcgplayerPriceGuide.lua to set OFFLINE=true (don't forget to change it back once you're done and want live data again)
Now for the namereplacement
  • The target names can be found in lib\LHpi.Data-v5.lua (for cards that have variants) or in MA (Oracle name or localized name)
  • To find the source names, set VERBOSE=true and LOGNAMEREPLACE=true (I'd set LOGDROPS,LOGFOILTWEAK,STRICTEXPECTED true as well),
  • then run the script again and open the log
  • find all lines that start with "! LHpi.SetPrice". those tell you exactly which cards failed to be set, including the name LHpi sent to MA. For tokens, suffix the name with " Token" to enable LHpi to detect the object type.
  • If the site_name-to-LHpi_name mapping is not obvious, you can compare the card image on the site with the one in MA (prominent example: Basic Lands in sets without collector numbers).
  • Run the script again and observe the new "namereplaced X to Y" lines in the log, then hopefully see how the number of failed cards has decreased, while the number of namerepalced cards has increased.
  • repeat until either all cards from the source are identified or all cards in MA have received the correct price
  • change site.expected to tell LHpi how many successfully set ("pset"), failed and namereplaced cards to expect.

EDIT: download links on wiki page changed to corrected version
EDIT2: if the above heuristic fails, you can either try to figure it out by looking at the other sets' namereplacement and variant tables, or report back to me and I will fix the rest.
EDIT3: best do the download run from MA. If you use the dummyMA in OFFLINE=false mode, you either need to provide luasockets yourself or wait until I get around documenting installation and license for the external dependencies.
EDIT4: The options described in the above steps are now much more sensible... please read the code comments to sanity check anything I say in this post ;-)
EDIT5: If you have cards that refuse to be namereplaced and your suspect that special characters are to blame, you can try writing them as "\ddd" (where ddd is the byte value, padded with leading zeros to three digits). Lua is character encoding agnistic, every string is just a string of bytes internally.
  • set DEBUG=true
  • The log is now very wordy, but you can search for (part of) the card name.
  • At some point, there will be a byte-for-byte representation of the name in the log.
  • This is the state of the card name manipulation directly before LHpi looks for a namereplace entry (which, in turn, happens before foiltweak, variant and object type detecion.)
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

Origins for everyone :)

Postby LordHelmchen » 21 Jul 2015, 21:13

Here's the promised update of all existing sitescripts to 2.15.6

All sets up to [822] (Magic Origins) are included, as far as a source url could be identified. Some unsorted promo sets may still be commented out.
As announced, I have neither resolved problematic cards (requires updating/adding namereplacement tables, and potentially other tricks for complicated corner cases), nor updated site.expected tables to let the script know exactly how many set/failed/dropped/namereplaced/foiltweaked cards are needed to consider the set imported perfectly.
In addition, all scripts now
  • respond properly to site.Initialize("update") or site.Initialize({ update=true })
  • respond properly to site.BuildUrl("list"),
  • have site.updateFormatString defined and adjusted
  • implement site.FetchExpansionList()
Therefore, in the future, if you want to maintain one of the sitecripts, you can use the dummyMA.lua to run the sitescript in update helper mode which makes adding missing sets to the sitescript a breeze. Resolving missed/failed cards still requires manually looking for the right namereplacement, but the bulk of cards will probably work out-of-the-box.
external dependencies.zip
only needed if you use the dummyMA
(203.32 KiB) Downloaded 157 times


Last test run before zip'ing up showed the following numbers:
Code: Select all
[822] (Magic Origins) contains  286 cards ( 272 regular,   14 tokens,    0 nontraditional,    0 replica )
LHpi.tcgplayerPriceGuide-v2.15.6.14.log | Open
Code: Select all
 :-) Prices for set Magic Origins (id 822) were imported as expected :-)
Total counted : 27197 set, 171 failed English cards   1 set, 0 failed Arabic cards   1 set, 0 failed Latin cards   1 set, 0 failed Hebrew cards   1 set, 0 failed Phyrexian cards   1 set, 0 failed Ancient Greek cards   1 set, 0 failed Sanskrit cards   49 set, 1 failed Japanese cards   ; 164 dropped, 992 namereplaced and 221 foiltweaked.
Total expected: 27197 set, 171 failed English cards   1 set, 0 failed Arabic cards   1 set, 0 failed Latin cards   1 set, 0 failed Hebrew cards   1 set, 0 failed Phyrexian cards   1 set, 0 failed Ancient Greek cards   1 set, 0 failed Sanskrit cards   49 set, 1 failed Japanese cards   ; 164 dropped, 985 namereplaced and 222 foiltweaked.
count differs in 0 sets: { }
LHpi.mtgmintcard.lua.zip
(14.02 KiB) Downloaded 173 times
LHpi.mtgmintcard-v2.15.6.14.log | Open
Code: Select all
 :-( persetcount for Magic Origins (id 822) differs from expected.
 :-( counted : 264 set &  37 failed  English cards ; 255 set &  46 failed Simplified Chinese cards ;   0 dropped,   0 namereplaced and   0 foiltweaked.
 :-( expected: 286 set &   0 failed  English cards ; 272 set &   0 failed Simplified Chinese cards ;   0 dropped,   0 namereplaced and   0 foiltweaked.
Total counted : 24130 set, 128 failed English cards   8217 set, 307 failed Simplified Chinese cards   ; 850 dropped, 136 namereplaced and 46 foiltweaked.
Total expected: 24352 set, 67 failed English cards   8780 set, 243 failed Simplified Chinese cards   ; 813 dropped, 137 namereplaced and 39 foiltweaked.
count differs in 22 sets: { [390]="Starter 1999";[570]="Darksteel";[630]="9th Edition";[700]="Planar Chaos";[740]="Duel Decks: Elves vs. Goblins";[755]="Duel Decks: Jace vs. Chandra";[757]="Duel Decks: Divine vs. Demonic";[758]="Alara Reborn";[762]="Zendikar";[763]="Duel Decks: Garruk vs. Liliana";[778]="Magic: The Gathering Commander";[784]="Dark Ascension";[786]="Avacyn Restored";[800]="Theros";[808]="Magic 2015";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[819]="Modern Masters 2015 Edition";[820]="Duel Decks: Elspeth vs. Kiora";[822]="Magic Origins";}
LHpi.mtgprice.com.lua.zip
(14.34 KiB) Downloaded 184 times
LHpi.mtgprice.com-v2.15.6.5.log | Open
Code: Select all
 :-( persetcount for Magic Origins (id 822) differs from expected.
 :-( counted : 225 set &  10 failed  English cards ;   0 dropped,   0 namereplaced and   0 foiltweaked.
 :-( expected: 272 set &   0 failed  English cards ;   0 dropped,   0 namereplaced and   0 foiltweaked.
Total counted : 25436 set, 610 failed English cards   ; 0 dropped, 311 namereplaced and 150 foiltweaked.
Total expected: 26027 set, 142 failed English cards   ; 0 dropped, 380 namereplaced and 298 foiltweaked.
count differs in 56 sets: { [5]="Book Inserts";[6]="Comic Inserts";[7]="Magazine Inserts";[9]="Video Game Promos";[10]="Junior Series Promos";[21]="Release & Launch Parties Promos";[22]="Prerelease Promos";[23]="Gateway & WPN Promos";[25]="Judge Gift Cards";[26]="Magic Game Day";[27]="Alternate Art Lands";[30]="Friday Night Magic Promos";[40]="Arena/Colosseo Leagues Promos";[42]="Summer of Magic Promos";[120]="Arabian Nights";[160]="The Dark";[190]="Ice Age";[200]="Chronicles";[201]="Renaissance";[225]="Introductory Two-Player Set";[260]="Portal";[270]="Weatherlight";[320]="Unglued";[380]="Portal Three Kingdoms";[390]="Starter 1999";[415]="Starter 2000";[450]="Planeshift";[490]="Deckmasters";[550]="8th Edition";[600]="Unhinged";[630]="9th Edition";[670]="Coldsnap";[680]="Time Spiral";[690]="Time Spiral Timeshifted";[756]="Conflux";[769]="Archenemy";[776]="New Phyrexia";[782]="Innistrad";[784]="Dark Ascension";[787]="Planechase 2012 Edition";[791]="Return to Ravnica";[793]="Gatecrash";[794]="Duel Decks: Sorin vs. Tibalt";[799]="Duel Decks: Heroes vs. Monsters";[800]="Theros";[801]="Commander 2013 Edition";[805]="Duel Decks: Jace vs. Vraska";[807]="Conspiracy";[809]="From the Vault: Annihilation";[812]="Duel Decks: Speed vs. Cunning";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[820]="Duel Decks: Elspeth vs. Kiora";[822]="Magic Origins";}
LHpi.magicuniverseDE.lua.zip
(15.57 KiB) Downloaded 191 times
LHpi.magicuniverseDE-v2.15.6.14.log | Open
Code: Select all
 :-( persetcount for Magic Origins (id 822) differs from expected.
 :-( counted : 267 set &   5 failed  English cards ; 267 set &   5 failed   German cards ;   0 dropped,   0 namereplaced and   0 foiltweaked.
 :-( expected: 286 set &   0 failed  English cards ; 286 set &   0 failed   German cards ;   0 dropped,   0 namereplaced and   0 foiltweaked.
Total counted : 19607 set, 457 failed English cards   17544 set, 492 failed German cards   1 set, 0 failed French cards   19 set, 0 failed Italian cards   ; 1102 dropped, 190 namereplaced and 19 foiltweaked.
Total expected: 19630 set, 442 failed English cards   17926 set, 480 failed German cards   1 set, 0 failed French cards   19 set, 0 failed Italian cards   ; 1132 dropped, 191 namereplaced and 15 foiltweaked.
count differs in 27 sets: { [139]="Revised Edition (FBB)";[150]="Legends";[160]="The Dark";[180]="4th Edition";[190]="Ice Age";[220]="Alliances";[230]="Mirage";[280]="Tempest";[300]="Exodus";[330]="Urza’s Saga";[400]="Mercadian Masques";[410]="Nemesis";[420]="Prophecy";[430]="Invasion";[460]="7th Edition";[470]="Apocalypse";[480]="Odyssey";[520]="Onslaught";[560]="Mirrodin";[590]="Champions of Kamigawa";[670]="Coldsnap";[800]="Theros";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[822]="Magic Origins";}
LHpi.trader-onlineDE.lua.zip
(16.57 KiB) Downloaded 177 times
LHpi.trader-onlineDE-v2.15.6.13.log | Open
Code: Select all
 :-( persetcount for Magic Origins (id 822) differs from expected.
 :-( counted : 281 set &   5 failed  English cards ; 270 set &  16 failed   German cards ;   0 dropped,   0 namereplaced and   0 foiltweaked.
 :-( expected: 286 set &   0 failed  English cards ; 286 set &   0 failed   German cards ;   0 dropped,   0 namereplaced and   0 foiltweaked.
Total counted : 22346 set, 962 failed English cards   19547 set, 117 failed German cards   429 set, 0 failed Italian cards   ; 1314 dropped, 333 namereplaced and 11 foiltweaked.
Total expected: 23069 set, 40 failed English cards   20536 set, 53 failed German cards   429 set, 0 failed Italian cards   ; 1332 dropped, 339 namereplaced and 11 foiltweaked.
count differs in 12 sets: { [100]="Beta";[105]="Collectors’ Edition (Domestic)";[106]="Collectors’ Edition (International)";[110]="Unlimited";[800]="Theros";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[819]="Modern Masters 2015 Edition";[820]="Duel Decks: Elspeth vs. Kiora";[822]="Magic Origins";}
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

Release tcgplayerPriceGuide 2.15.7.15 (BFZ)

Postby LordHelmchen » 28 Sep 2015, 00:31

While I'm waiting for the total lunar eclipse, I finished the LHpi.tcgplayerPriceGuide.lua BFZ update :)
Barring bug 754, all sets should import as expected.
Also, at last a set release without changes to the library, so only a new LHpi.Data is needed this time :mrgreen:

EDIT: and once again, forgot to comment out all non-default options. edit yourself or redownload. Sorry.
Attachments
LHpi.tcgplayerPriceGuide.lua.zip
(28.08 KiB) Downloaded 68 times
LHpi.Data-v7.lua.zip
(41.96 KiB) Downloaded 184 times
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

Update 2.16.8

Postby LordHelmchen » 08 Oct 2015, 14:18

As usual, all download links and documentation are at LHpi's wiki page.
I also added the insgtructions from the previous posts to http://www.slightlymagic.net/wiki/How-To_sitescriptTemplate#Updating_sitescripts.
If you're only here for LHpi.tcgplayerPriceGuide, you can ignore the next post :-)
Attachments
LHpi.sitescriptTemplate-v2.16.8.14.lua.zip
(13.67 KiB) Downloaded 129 times
LHpi.Data-v8.lua.zip
finally all sets have variant tables
(55.82 KiB) Downloaded 168 times
LHpi-v2.16.lua.zip
(19.53 KiB) Downloaded 289 times
LHpi.tcgplayerPriceGuide.lua.zip
version 2.16.8.16
(28.24 KiB) Downloaded 276 times
Last edited by LordHelmchen on 08 Oct 2015, 14:27, edited 1 time in total.
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

LHpi.magickartenmarkt

Postby LordHelmchen » 08 Oct 2015, 14:20

LHpi.magickartenmarkt is finally ready to be tested by a wider audience.
Due to the limitations imposed by Magic Album's embedded Lua interpreter and the MKM API, this is not as straight forward as the other sitescripts. I recommend you take the time to read the additional instructions for MKM.
Attachments
LHpi.external_dependencies-v2.zip
needed for dummy and mkm
(939.35 KiB) Downloaded 197 times
LHpi.dummyMA.lua.zip
version 0.7
(11.04 KiB) Downloaded 115 times
LHpi.mkm-helper.lua.zip
version 2.16.8.1
(8.23 KiB) Downloaded 121 times
LHpi.magickartenmarkt.lua.zip
version 2.16.8.2
(41.78 KiB) Downloaded 113 times
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

LHpi.mkm update 2.16.9

Postby LordHelmchen » 14 Oct 2015, 10:57

As they say: release early, release often.
Most important changes in this version (see wiki for full changelog):
Improved http error handling by site.FetchSourceDataFromOauth in LHpi.magickartenmarkt and mkm-helper.
LHpi.mkm-helper now gracefully skips the current set on http errors (including 429 - Too Many Requests, which can happen often in the initial learning phase).
All functions in LHpi.dummyMA now consistently start with an upper case letter. Calls in other scripts were changed.
LHpi.Data only received a minor fix/update (cardcount for 26).

If you're a seller on MKM and you're feeling adventurous (only GET methods used, your stock should be in no danger 8) ), you might be interested to check out the new experimental, unpolished and largely undocumented checkstock mode in mkm-helper.

EDIT:
checkstock mode was called checksales during early development. I may have changed the documentation, but not the code that checks for detected options. Please try the other if one does not do anything. Will be corrected in next version, but it's not worth releasing a new version yet.
/EDIT
Attachments
LHpi.magickartenmarkt.lua.zip
version 2.16.9.3
(42.03 KiB) Downloaded 116 times
LHpi.mkm-helper.lua.zip
version 2.16.9.2
(10.63 KiB) Downloaded 133 times
LHpi.dummyMA.lua.zip
version 0.8
needed by mkm-helper 2.16.9.2
(11.02 KiB) Downloaded 186 times
LHpi.Data-v9.lua.zip
only minor change from v8
(55.71 KiB) Downloaded 278 times
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

Re: LordHelmchen's price import - BFZ and mkm

Postby LordHelmchen » 18 Oct 2015, 09:55

Small update:
timestamp for last fetch of offline data is only updated when no errors were encountered.
timestamp and requestCounter file are properly initialized if none exist (Thanks to Danatar for reporting and testing)

I apologize for the inconveniance. I really should have deleted my local cache and tested if it works on an fresh install.
Attachments
LHpi.magickartenmarkt.lua.zip
version 2.16.9.4
(42.04 KiB) Downloaded 263 times
LHpi.mkm-helper.lua.zip
version 2.16.9.3
(10.78 KiB) Downloaded 236 times
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

Possible TOS problem with MKM

Postby LordHelmchen » 09 Dec 2015, 17:12

I just received notice that at least one User of my scripts has had their mkm tokens revoked due to using my script.

It would probably be prudent to update only a few sets per day, and to stay below the 5000 requests limit for the time being.
LordHelmchen
 
Posts: 125
Joined: 21 Aug 2012, 16:06
Has thanked: 20 times
Been thanked: 32 times

PreviousNext

Return to Magic Album

Who is online

Users browsing this forum: No registered users and 4 guests


Who is online

In total there are 4 users online :: 0 registered, 0 hidden and 4 guests (based on users active over the past 10 minutes)
Most users ever online was 279 on 11 Jul 2013, 22:03

Users browsing this forum: No registered users and 4 guests

Login Form