Difference between revisions of "LHpi"
LordHelmchen (Talk | contribs) (2.12.4 released) |
LordHelmchen (Talk | contribs) (mkm 2.16.9.4 and helper 2.16.9.3) |
||
(30 intermediate revisions by one user not shown) | |||
Line 1: | Line 1: | ||
LHpi is LordHelmchen's price import library and sitescripts for Magic Album. | LHpi is LordHelmchen's price import library and sitescripts for Magic Album. | ||
− | LHpi is a collection of lua scripts to be used by Magic Album, but is '''''not''' part of Magic Album and thus updated and supported | + | LHpi is a collection of lua scripts to be used by Magic Album, but is '''''not''' part of Magic Album and thus updated and supported separately''. |
Discuss, ask for help and leave feedback at the [http://www.slightlymagic.net/forum/viewtopic.php?f=32&t=8174 corresponding Forum Thread]. | Discuss, ask for help and leave feedback at the [http://www.slightlymagic.net/forum/viewtopic.php?f=32&t=8174 corresponding Forum Thread]. | ||
==Sitescripts== | ==Sitescripts== | ||
− | The sitescripts are the actual price import scripts, written in lua. They use the [[Magic Album Lua API]] and [[# | + | The sitescripts are the actual price import scripts, written in lua. They use the [[Magic Album Lua API]] and [[#the LHpi Library]] to query and parse a website and import the price information into Magic Album. Most of the program logic is contained in the library, so the sitescript needs to hold only site-specific information and a few functions that are not easily generalizable. |
In theory, you should not be dependent on a new version once a new set is available: just add new entries to site.sets table in the sitescript. Being easy to update yourself was one of the design goals. | In theory, you should not be dependent on a new version once a new set is available: just add new entries to site.sets table in the sitescript. Being easy to update yourself was one of the design goals. | ||
Sitescripts will need to provide | Sitescripts will need to provide | ||
− | * a table of supported sets with language and | + | * a table of supported sets with language and URL infix fields |
− | * a table of supported languages with | + | * a table of supported languages with URL infix field |
− | * a function to build | + | * a function to build URL |
− | * a function to parse the | + | * a function to parse the HTML raw data into data fields via regular expressions (and the regex itself) |
* optionally, they can also provide | * optionally, they can also provide | ||
** functions for site specific data manipulation, in case the site has some strange formatting the library does not understand | ** functions for site specific data manipulation, in case the site has some strange formatting the library does not understand | ||
Line 19: | Line 19: | ||
** tables to map card names to picture versions, in case the one-to-one map and LHpi's defaults are not sufficient | ** tables to map card names to picture versions, in case the one-to-one map and LHpi's defaults are not sufficient | ||
** a table with expected card count numbers for LHpi's sanity checks. | ** a table with expected card count numbers for LHpi's sanity checks. | ||
− | === | + | ** a function to fetch a list of expansions from the site, to be used by update helper functions. |
+ | |||
+ | |||
+ | ===magickartenmarkt=== | ||
{| | {| | ||
− | |[http://www.slightlymagic.net/forum/download/file.php?id= | + | |[http://www.slightlymagic.net/forum/download/file.php?id=23445 LHpi.magickartenmarkt.lua.zip] |
|- | |- | ||
− | |Sitescript version||2. | + | |Sitescript version||2.16.9.4 |
|- | |- | ||
− | |updated last on|| | + | |updated last on||2015-10-19 |
|- | |- | ||
− | |size|| | + | |size||42.04 KiB |
|} | |} | ||
− | [ | + | [https://www.magickartenmarkt.de/ MagicKartenMarkt] (or [https://www.magiccardmarket.eu/ magiccardmarket] is "Europe's largest online marketplace for Magic the Gathering cards!". It has become the de facto standard way to acquire singles, at least in my local MtG circles. |
− | + | <s>They are the only price source that I personally am interested in anymore, but the script is not in a releasable state yet. See the forum thread, starting at [http://www.slightlymagic.net/forum/viewtopic.php?f=32&t=8174&start=45#p182474 state of things] for details on how far along development is.</s> Due to the limitations imposed by Magic Album's embedded Lua interpreter and the MKM API, you cannot fetch live data with the sitescript. You can use LHpi.mkm-helper.lua to download price data. | |
− | === | + | Additional preparation steps are needed for LHpi.magickartenmarkt. Please refer to the [[#additional instructions for MKM]] below. |
+ | |||
+ | ====mkm-helper.lua==== | ||
{| | {| | ||
− | |[http://www.slightlymagic.net/forum/download/file.php?id= | + | |[http://www.slightlymagic.net/forum/download/file.php?id=23444 LHpi.mkm-helper.zip] |
|- | |- | ||
− | |Sitescript version||2. | + | |Sitescript version||2.16.9.3 |
|- | |- | ||
− | | | + | |recommended dummyMA version||>=0.8 |
|- | |- | ||
− | |size|| | + | |updated last on||2015-10-19 |
+ | |- | ||
+ | |size||10.78 KiB | ||
|} | |} | ||
− | + | Helper script to download price data from MKM. The data can then be imported to MA with magickartenmarkt.lua. | |
− | + | The MKM API imposes a limit of 5000 request per day. With over 30000 English cards known to Magic Album, mkm-helper tries to help space out the download over several days and keeps a record of when a set was last downloaded.<br /> | |
+ | Also included is a batch file (LHpi.mkm-helper.bat) for ease of use. Edit it to point to your MA\Prices directory and provide a list of sets you want downloaded. | ||
− | === | + | ===tcgplayer.com PriceGuide=== |
{| | {| | ||
− | |[http://www.slightlymagic.net/forum/download/file.php?id= | + | |[http://www.slightlymagic.net/forum/download/file.php?id=23327 LHpi.tcgplayerPriceGuide.lua.zip] |
|- | |- | ||
− | |Sitescript version||2. | + | |Sitescript version||2.16.8.16 |
|- | |- | ||
− | |updated last on|| | + | |updated last on||2015-10-08 |
|- | |- | ||
− | |size|| | + | |size||28.24 KiB |
|} | |} | ||
− | [http://magic.tcgplayer.com/magic_price_guides.asp Tcgplayer Price Guide] claims to accumulate prices from 75 online vendors. They are priced in $. Their tables give a high, low and average price; the sitescript contains a variable to select which one of those it should use. | + | [http://magic.tcgplayer.com/magic_price_guides.asp Tcgplayer Price Guide] claims to accumulate prices from 75 online vendors. They are priced in $. |
− | + | As requested by Golob [http://www.slightlymagic.net/forum/viewtopic.php?f=32&t=2324&start=300#p108605 here].<br /> | |
− | <br /> | + | |
− | + | Their tables give a high, low and average price; the sitescript contains a variable to select which one of those it should use. | |
− | + | To change the column to import from, open the script in a text editor, find <code>himelo = 3</code> and change it to <code>himelo = 1</code> for high or <code>himelo = 2</code> for medium prices. You can have [[How-To_sitescriptTemplate#multiple_copies_ot_the_same_script|multiple copies of the same script]]<br /> | |
− | Lately, their version handling has somewhat improved, but there's still a lot of guesswork involved... | + | Note that cards with multiple versions are averaged in some sets and set to "variant 1"'s price in others, as the source list does not clearly identify the version. Even when it does, it is unclear if the entry without version suffix is meant to be all unspecified or version 1 only. In most cases, "CARD" price is set for all versions and averaged with "CARD (version)" price for given versions now. Lately, their version handling has somewhat improved, but there's still a lot of guesswork involved... |
+ | |||
+ | ===magicuniverse.de=== | ||
+ | {| | ||
+ | |[http://www.slightlymagic.net/forum/download/file.php?id=22775 LHpi.magicuniverseDE.lua.zip] | ||
+ | |- | ||
+ | |Sitescript version||2.15.6.14 | ||
+ | |- | ||
+ | |updated last on||2015-07-21 | ||
+ | |- | ||
+ | |size||15.57 KiB | ||
+ | |} | ||
+ | [http://www.magicuniverse.de/html/einzelkarten.php?menue Magic Universe] is a German site with German and English cards, priced in €. | ||
+ | |||
+ | This site started it all, since I wanted prices in local currency, and my FLG apparently used them as price reference. | ||
+ | |||
+ | MAINTAINER WANTED! | ||
+ | I do not need any prices besides from MKM, and will not update this (unless specifically requested) anymore. The last test results were: | ||
+ | <tt> | ||
+ | :-( persetcount for Magic Origins (id 822) differs from expected. | ||
+ | :-( counted : 267 set & 5 failed English cards ; 267 set & 5 failed German cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. | ||
+ | :-( expected: 286 set & 0 failed English cards ; 286 set & 0 failed German cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. | ||
+ | Total counted : 19607 set, 457 failed English cards 17544 set, 492 failed German cards 1 set, 0 failed French cards 19 set, 0 failed Italian cards ; 1102 dropped, 190 namereplaced and 19 foiltweaked. | ||
+ | Total expected: 19630 set, 442 failed English cards 17926 set, 480 failed German cards 1 set, 0 failed French cards 19 set, 0 failed Italian cards ; 1132 dropped, 191 namereplaced and 15 foiltweaked. | ||
+ | count differs in 27 sets: { [139]="Revised Edition (FBB)";[150]="Legends";[160]="The Dark";[180]="4th Edition";[190]="Ice Age";[220]="Alliances";[230]="Mirage";[280]="Tempest";[300]="Exodus";[330]="Urza’s Saga";[400]="Mercadian Masques";[410]="Nemesis";[420]="Prophecy";[430]="Invasion";[460]="7th Edition";[470]="Apocalypse";[480]="Odyssey";[520]="Onslaught";[560]="Mirrodin";[590]="Champions of Kamigawa";[670]="Coldsnap";[800]="Theros";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[822]="Magic Origins";} | ||
+ | </tt> | ||
+ | |||
+ | ===mtgmintcard.com=== | ||
+ | {| | ||
+ | |[http://www.slightlymagic.net/forum/download/file.php?id=22773 LHpi.mtgmintcard.lua.zip] | ||
+ | |- | ||
+ | |Sitescript version||2.15.6.14 | ||
+ | |- | ||
+ | |updated last on||2015-07-21 | ||
+ | |- | ||
+ | |size||14.02 KiB | ||
+ | |} | ||
+ | Functionally the same as [[#Other scripts|#"MTG Mint Card.lua"]] by Goblin Hero and Stromglad1. | ||
+ | |||
+ | When I wanted to generalize my script for easy adaption to other sites, this provided me with a good baseline, since I could search MA for cards that my script missed but should have set. Also, the site is easy to query and parse. They do have a currency chooser, so I started by trying to make a mtgmintcard-Euro script. I failed, but salvaged the wreck for default $ prices. | ||
+ | |||
+ | MAINTAINER WANTED! | ||
+ | I do not need any prices besides from MKM, and will not update this (unless specifically requested) anymore. The last test results were: | ||
+ | <tt> | ||
+ | :-( persetcount for Magic Origins (id 822) differs from expected. | ||
+ | :-( counted : 264 set & 37 failed English cards ; 255 set & 46 failed Simplified Chinese cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. | ||
+ | :-( expected: 286 set & 0 failed English cards ; 272 set & 0 failed Simplified Chinese cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. | ||
+ | Total counted : 24130 set, 128 failed English cards 8217 set, 307 failed Simplified Chinese cards ; 850 dropped, 136 namereplaced and 46 foiltweaked. | ||
+ | Total expected: 24352 set, 67 failed English cards 8780 set, 243 failed Simplified Chinese cards ; 813 dropped, 137 namereplaced and 39 foiltweaked. | ||
+ | count differs in 22 sets: { [390]="Starter 1999";[570]="Darksteel";[630]="9th Edition";[700]="Planar Chaos";[740]="Duel Decks: Elves vs. Goblins";[755]="Duel Decks: Jace vs. Chandra";[757]="Duel Decks: Divine vs. Demonic";[758]="Alara Reborn";[762]="Zendikar";[763]="Duel Decks: Garruk vs. Liliana";[778]="Magic: The Gathering Commander";[784]="Dark Ascension";[786]="Avacyn Restored";[800]="Theros";[808]="Magic 2015";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[819]="Modern Masters 2015 Edition";[820]="Duel Decks: Elspeth vs. Kiora";[822]="Magic Origins";} | ||
+ | </tt> | ||
===trader-online.de=== | ===trader-online.de=== | ||
{| | {| | ||
− | |[http://www.slightlymagic.net/forum/download/file.php?id= | + | |[http://www.slightlymagic.net/forum/download/file.php?id=22772 LHpi.trader-onlineDE.lua.zip] |
|- | |- | ||
− | |Sitescript version||2. | + | |Sitescript version||2.15.6.14 |
|- | |- | ||
− | |updated last on|| | + | |updated last on||2015-07-21 |
|- | |- | ||
− | |size|| | + | |size||16.57 KiB |
|} | |} | ||
[http://www.trader-online.de/Magic_Online-Shop.php Trader Online] is a German site with German and English cards, priced in €. | [http://www.trader-online.de/Magic_Online-Shop.php Trader Online] is a German site with German and English cards, priced in €. | ||
− | I found this site while looking for Token images, found it to be | + | I found this site while looking for Token images, found it to be parsable by sitescript, and decided to use it as a test on how well the library was adapted to support additional sites. |
+ | |||
+ | MAINTAINER WANTED! | ||
+ | I do not need any prices besides from MKM, and will not update this (unless specifically requested) anymore. The last test results were: | ||
+ | <tt> | ||
+ | :-( persetcount for Magic Origins (id 822) differs from expected. | ||
+ | :-( counted : 281 set & 5 failed English cards ; 270 set & 16 failed German cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. | ||
+ | :-( expected: 286 set & 0 failed English cards ; 286 set & 0 failed German cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. | ||
+ | Total counted : 22346 set, 962 failed English cards 19547 set, 117 failed German cards 429 set, 0 failed Italian cards ; 1314 dropped, 333 namereplaced and 11 foiltweaked. | ||
+ | Total expected: 23069 set, 40 failed English cards 20536 set, 53 failed German cards 429 set, 0 failed Italian cards ; 1332 dropped, 339 namereplaced and 11 foiltweaked. | ||
+ | count differs in 12 sets: { [100]="Beta";[105]="Collectors’ Edition (Domestic)";[106]="Collectors’ Edition (International)";[110]="Unlimited";[800]="Theros";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[819]="Modern Masters 2015 Edition";[820]="Duel Decks: Elspeth vs. Kiora";[822]="Magic Origins";} | ||
+ | </tt> | ||
===mtgprice.com=== | ===mtgprice.com=== | ||
{| | {| | ||
− | |[http://www.slightlymagic.net/forum/download/file.php?id= | + | |[http://www.slightlymagic.net/forum/download/file.php?id=22774 LHpi.mtgprice.com.lua.zip] |
|- | |- | ||
− | |Sitescript version||2. | + | |Sitescript version||2.15.6.5 |
|- | |- | ||
− | |updated last on|| | + | |updated last on||2015-07-21 |
|- | |- | ||
− | |size|| | + | |size||14.34 KiB |
|} | |} | ||
I did not plan to include it in the sitescript collection, but now that the cat is out of the bag and it has been downloaded a dual-digit number of times, I can at least upload the newer version to make it compatible with the latest lib and data... | I did not plan to include it in the sitescript collection, but now that the cat is out of the bag and it has been downloaded a dual-digit number of times, I can at least upload the newer version to make it compatible with the latest lib and data... | ||
Contains a variable to select between "Fair Trade" and "best Buy" price column. | Contains a variable to select between "Fair Trade" and "best Buy" price column. | ||
− | It still needs a lot of namereplace tables. I'm not that much interested in $ prices, so if you are, consider volunteering to finish and maintain it. | + | It still needs a lot of namereplace tables. I'm not that much interested in $ prices, so if you are, consider volunteering to finish and maintain it. Post in the thread if you want to do it but are unsure what to do. |
− | ==The LHpi Library== | + | MAINTAINER WANTED! |
+ | I do not need any prices besides from MKM, and will not update this anymore (unless specifically requested). The last test results were: | ||
+ | <tt> | ||
+ | :-( persetcount for Magic Origins (id 822) differs from expected. | ||
+ | :-( counted : 225 set & 10 failed English cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. | ||
+ | :-( expected: 272 set & 0 failed English cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. | ||
+ | Total counted : 25436 set, 610 failed English cards ; 0 dropped, 311 namereplaced and 150 foiltweaked. | ||
+ | Total expected: 26027 set, 142 failed English cards ; 0 dropped, 380 namereplaced and 298 foiltweaked. | ||
+ | count differs in 56 sets: { [5]="Book Inserts";[6]="Comic Inserts";[7]="Magazine Inserts";[9]="Video Game Promos";[10]="Junior Series Promos";[21]="Release & Launch Parties Promos";[22]="Prerelease Promos";[23]="Gateway & WPN Promos";[25]="Judge Gift Cards";[26]="Magic Game Day";[27]="Alternate Art Lands";[30]="Friday Night Magic Promos";[40]="Arena/Colosseo Leagues Promos";[42]="Summer of Magic Promos";[120]="Arabian Nights";[160]="The Dark";[190]="Ice Age";[200]="Chronicles";[201]="Renaissance";[225]="Introductory Two-Player Set";[260]="Portal";[270]="Weatherlight";[320]="Unglued";[380]="Portal Three Kingdoms";[390]="Starter 1999";[415]="Starter 2000";[450]="Planeshift";[490]="Deckmasters";[550]="8th Edition";[600]="Unhinged";[630]="9th Edition";[670]="Coldsnap";[680]="Time Spiral";[690]="Time Spiral Timeshifted";[756]="Conflux";[769]="Archenemy";[776]="New Phyrexia";[782]="Innistrad";[784]="Dark Ascension";[787]="Planechase 2012 Edition";[791]="Return to Ravnica";[793]="Gatecrash";[794]="Duel Decks: Sorin vs. Tibalt";[799]="Duel Decks: Heroes vs. Monsters";[800]="Theros";[801]="Commander 2013 Edition";[805]="Duel Decks: Jace vs. Vraska";[807]="Conspiracy";[809]="From the Vault: Annihilation";[812]="Duel Decks: Speed vs. Cunning";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[820]="Duel Decks: Elspeth vs. Kiora";[822]="Magic Origins";} | ||
+ | </tt> | ||
+ | |||
+ | ==the LHpi Library== | ||
{| | {| | ||
− | |[http://www.slightlymagic.net/forum/download/file.php?id= | + | |[http://www.slightlymagic.net/forum/download/file.php?id=23328 LHpi-v2.16.lua.zip] |
+ | |- | ||
+ | |Tested with Ma version||1.6.4.348 | ||
+ | |- | ||
+ | |recommended Data version||>=8 | ||
+ | |- | ||
+ | |updated last on||2015-10-08 | ||
+ | |- | ||
+ | |size||19.53 KiB | ||
+ | |} | ||
+ | {| | ||
+ | |[http://www.slightlymagic.net/forum/download/file.php?id=22737 LHpi-v2.15.lua.zip] | ||
+ | |- | ||
+ | |Tested with Ma version||1.6.4.348 | ||
|- | |- | ||
− | | | + | |recommended Data version|| >=6 |
|- | |- | ||
− | |updated last on|| | + | |updated last on||2015-07-18 |
|- | |- | ||
− | |size|| | + | |size||21 KiB |
|} | |} | ||
It is not expected to be fast, but as reliable as possible:<br /> | It is not expected to be fast, but as reliable as possible:<br /> | ||
Line 106: | Line 199: | ||
Then it loops though the sets that are both available and selected:<br /> | Then it loops though the sets that are both available and selected:<br /> | ||
The raw html data is fetched, parsed and entered into an intermediate table of all prices for the current edition, merging regular and foil prices to minimize calls to ma.SetPrice.<br /> | The raw html data is fetched, parsed and entered into an intermediate table of all prices for the current edition, merging regular and foil prices to minimize calls to ma.SetPrice.<br /> | ||
− | Should the same card (defined by set,card,language,(picture-)version ) be encountered again, it compares the prices. If they are equal, the new price is discarded; if they differ | + | Should the same card (defined by set,card,language,(picture-)version ) be encountered again, it compares the prices. If they are equal, the new price is discarded; if they differ, it uses the average price. Cards that are suffxed with a condition description are currently disregarded.<br /> |
Once that cardsetTable is completed (all urls belonging to the set are parsed), it sends the prices to MA. | Once that cardsetTable is completed (all urls belonging to the set are parsed), it sends the prices to MA. | ||
− | Finally, it compares the count of set prices with the number of cards in the set (or, if given, the expectation given in the sitescript). | + | Finally, it compares the count of set (and failed to be set) prices with the number of cards in the set (or, if given, the expectation given in the sitescript). |
See [[How-To_sitescriptTemplate#procedure_of_function_calls]] for details. | See [[How-To_sitescriptTemplate#procedure_of_function_calls]] for details. | ||
− | Logging to a seperate file is now disabled by default again (but still configurable by the variable SAVELOG in the script). The log (\Prices\LHpi.log) is overwritten on subsequent re-runs of | + | Logging to a seperate file is now disabled by default again (but still configurable by the variable SAVELOG in the script). The log (\Prices\LHpi.log) is overwritten on subsequent (re-)runs of any script. |
===LHpi.Data=== | ===LHpi.Data=== | ||
{| | {| | ||
− | |[http://www.slightlymagic.net/forum/download/file.php?id= | + | |[http://www.slightlymagic.net/forum/download/file.php?id=23389 LHpi.Data-v9.lua.zip] |
+ | |- | ||
+ | |Library version||>=2.15 | ||
+ | |- | ||
+ | |updated last on||2015-10-15 | ||
+ | |- | ||
+ | |size||55.71 KiB | ||
+ | |} | ||
+ | {| | ||
+ | |[http://www.slightlymagic.net/forum/download/file.php?id=23253 LHpi.Data-v7.lua.zip] | ||
|- | |- | ||
− | |Library version||>=2. | + | |Library version||>=2.15 |
|- | |- | ||
− | |updated last on|| | + | |updated last on||2015-09-28 |
|- | |- | ||
− | |size|| | + | |size|| 41.96 KiB |
|} | |} | ||
Since library version 2.7, site-independent set data (number of cards,foilage, default variant and foiltweak tables) were moved into a seperate data file that gets loaded by the library on initialization. This will allow to distinguish between code updates (to the library) and set data updates (which are usually needed when a new set is released). Version numbering of the sitescripts will reflect both library and data versions used. | Since library version 2.7, site-independent set data (number of cards,foilage, default variant and foiltweak tables) were moved into a seperate data file that gets loaded by the library on initialization. This will allow to distinguish between code updates (to the library) and set data updates (which are usually needed when a new set is released). Version numbering of the sitescripts will reflect both library and data versions used. | ||
==Installation and usage== | ==Installation and usage== | ||
− | #Download at least one sitescript and both the LHpi | + | # Download at least one sitescript and both [[#the LHpi Library]] and [[#LHpi.Data|data]] file in the version the sitescript uses. |
− | #* note the change to version numbers and file names in 2.9.2 , see [[# | + | #* note the change to version numbers and file names in 2.9.2 , see [[#version history]] for details. |
− | #Unzip into the "Prices" subdirectory of your Magic Album installation. | + | # Unzip into the "Prices" subdirectory of your Magic Album installation. |
− | #(recommended, but not yet strictly necessary) Create a subfolder called "lib" under "Prices" and move both library and data file into the subfolder. | + | # (<s>recommended, but not yet strictly necessary</s>) Create a subfolder called "lib" under "Prices" and move both library and data file into the subfolder. |
− | #(optional) If you want to use the SAVEHTML or SAVETABLE feature, you need to create an empty subdirectory, named like the sitescript without file extension (for example "Prices\LHpi.magicuniverseDE"). | + | # (optional) If you want to use the SAVEHTML or SAVETABLE feature, you need to create an empty subdirectory, named like the sitescript without file extension (for example "Prices\LHpi.magicuniverseDE"). |
− | #(re)start Magic Album and use the sitescript ('''not''' the library nor the data file. Step 3 makes this warning obsolete.) from Magic Album's Price Manager dialog. | + | # (re)start Magic Album and use the sitescript ('''not''' the library nor the data file. Step 3 makes this warning obsolete.) from Magic Album's Price Manager dialog. |
+ | ===additional instructions for MKM=== | ||
+ | # Please update [[#the LHpi Library]] to version >=2.16 and [[#LHpi.Data|data]] to >=8, if you have not done so yet. | ||
+ | # Step 3 from the basic instructions is not optional anymore. Library components need to be in the Prices\lib\ directory. | ||
+ | # Download the LHpi.magickartenmarkt [[#magickartenmarkt|#sitescript]] and [[#mkm-helper.lua|#helper]] and unzip into Prices\. | ||
+ | # Download [[#dummyMA]] (>=0.8) and [[#additional dependencies]] (>=2) and unzip into Prices\lib. Take care to keep the subdirectory structure of the dependencies intact. | ||
+ | # Obviously, we do need a place to store the downloaded price data, so as in step 4 above, create a subdirectory in Prices\. Unless you manually changed the scripts, it needs to be called "LHpi.magickartenmarkt" | ||
+ | # go to [https://www.magickartenmarkt.de/?mainPage=showMyAccount your MKM account page] and create a set of app security tokens. Then create a copy of lib\LHpi.mkm.tokens.example and edit it with your personal MKM tokens. '''Do not share this information with anyone!''' | ||
+ | #* LHpi only posts information retrieval requests through the MKM API, but I cannot accept any liability for anything that happens to your account. | ||
+ | #* If you're unsure whether to trust my scripts, please read LHpi's sourcecode before making a decision. :-) | ||
+ | #* Also, do note that the tokens are stored in a plaintext file on your computer (the one you create in this step) and, while LHpi.magickartenmarkt.lua or LHpi.mkm-helper are running, in memory. | ||
+ | # edit LHpi.magickartenmarkt.lua to change necessary settings: | ||
+ | #* change <code>local mkmtokenfile = "LHpi.mkm.tokens.example"</code> to the token file you created in step 5. | ||
+ | #* MKM provides multiple price types. Choose two and map them to MA's regular and foil price columns by editing <code> local useAsRegprice=5</code> and <code>local useAsFoilprice=6</code> (see details in the script's comments). | ||
+ | #* change <code>--OFFLINE = true</code> to <code>OFFLINE = true</code>. I left it in so LHpi.magickartenmarkt can display an error and mention LHpi.mkm-helper, in case you did not read this instructions ;-) | ||
+ | # (optional) edit LHpi.mkm-helper to set which sets should be downloaded by default (<code>MODE = { download=true, sets="standard" }</code>) and how long the data is valid (<code>local dataStaleAge = { ["default"] = 60*60*24*7, -- one week, ... }</code>) | ||
+ | # edit LHpi.mkm-helper.bat | ||
+ | #* change <code>cd /D D:\Magic - The Gathering\Magic Album\Prices</code> to point to your MA location. | ||
+ | #* In addition or alternative to MODE in LHpi.mkm-helper.lua, you can set download mode and sets to be downloaded as arguments in LHpi.mkm-helper.bat; you might want to change the preconfigured example set definitions to your personal need. | ||
+ | #* if you prefer commandline operation, you can also pass arguments to the batch file (as long as you keep the <code>%*</code> arguments intact when making changes). | ||
+ | #* MODE allows for one set definition, either a single string or a table. In the batch or as comand line argument, you can pass multiple set definitions. Valid strings are "core","expansion","special","promo", Magic Album set ids, set names or three letter codes. | ||
+ | #* If you want, you can create multiple copies of the batch file for easy access to different modes of operation. Due to the included <code>cd</code> instruction, those can be stored anywhere on your computer. | ||
+ | # Run LHpi.mkm-helper.bat to download pricing information. Then you can use LHpi.magickartenmarkt via Magic Album's Price Manager to import the downloaded data. | ||
+ | # At first, you will probably encounter HTTP 429 errors (too many requests). After each sucessfull download, the date and number of requests needed for the set will be recorded, and the errors will hopefully happen less frequent. | ||
+ | #* To reset the request counter, pass "resetcounter" argument to the script (via commandline or by editing the batch file). | ||
+ | |||
+ | ===Troubleshooting=== | ||
If something does not happen as expected | If something does not happen as expected | ||
* make sure the script is saved utf-8 encoded. It's customary for lua scripts to be saved ANSI encoded, but as MA expects utf input, this is what I considered the most wysiwyg approach to set cards with special characters in their names. | * make sure the script is saved utf-8 encoded. It's customary for lua scripts to be saved ANSI encoded, but as MA expects utf input, this is what I considered the most wysiwyg approach to set cards with special characters in their names. | ||
− | * open the lua file in a text editor and change "--VERBOSE = true" to "VERBOSE = true", likewise enable "LOGDROPS","LOGNAMEREPLACE","LOGFOILTWEAK"," | + | * open the lua file in a text editor and change "--VERBOSE = true" to "VERBOSE = true", likewise enable "LOGDROPS","LOGNAMEREPLACE","LOGFOILTWEAK","STRICTEXPECTED" and "SAVELOG". This will make the script provide more feedback to help track down any problems (or just to see what it was doing) and also enable additional checks. The logfile will be called "[MA Folder]\Prices\[scriptname].log" by unless manually configured otherwise. |
I'll try to answer any questions you might have. If the script misbehaves please follow the two steps above and attach a copy of the .log to your post. | I'll try to answer any questions you might have. If the script misbehaves please follow the two steps above and attach a copy of the .log to your post. | ||
Line 142: | Line 270: | ||
Detailed instructions on the properties and functions to define in your sitescript file can be found at [[How-To sitescriptTemplate]]. | Detailed instructions on the properties and functions to define in your sitescript file can be found at [[How-To sitescriptTemplate]]. | ||
+ | ====Hints for developers==== | ||
+ | Some information that might be useful if you want to write/improve sitescripts. Please add questions related to developing here, so I know what needs to be documented. | ||
− | ===sitescriptTemplate=== | + | moved to [[How-To_sitescriptTemplate#Hints_for_developing]] |
+ | |||
+ | ====sitescriptTemplate==== | ||
{| | {| | ||
− | |[http://www.slightlymagic.net/forum/download/file.php?id= | + | |[http://www.slightlymagic.net/forum/download/file.php?id=23334 LHpi.sitescriptTemplate-v2.16.8.14.lua.zip] |
|- | |- | ||
− | |Sitescript version||2. | + | |Sitescript version||2.16.8.14 |
|- | |- | ||
− | |updated last on|| | + | |updated last on||2015-10-08 |
|- | |- | ||
− | |size|| | + | |size||13.67 KiB |
|} | |} | ||
This is just an empty, commented template to help write new sitescripts. | This is just an empty, commented template to help write new sitescripts. | ||
− | ===dummyMA | + | ====dummyMA==== |
{| | {| | ||
− | |[http://www.slightlymagic.net/forum/download/file.php?id= | + | |[http://www.slightlymagic.net/forum/download/file.php?id=23390 LHpi.dummyMA.zip] |
|- | |- | ||
− | |version||0. | + | |version||0.8 |
|- | |- | ||
− | |updated last on|| | + | |updated last on||2015-10-14 |
|- | |- | ||
− | |size|| | + | |size||11.02 KiB |
|} | |} | ||
− | Magic Album API dummy to test and debug LHpi within an IDE (such as eclipse) and without needing Magic Album. | + | Magic Album API dummy to test and debug LHpi within an IDE (such as eclipse) and without needing Magic Album. |
+ | LHpi.mkm-helper needs this for ma.GetFile, ma.PutFile, dummy.mergetables and a few tables (corests,expansionsets,specialsets,promosets). | ||
− | ===Planned features=== | + | ====additional dependencies==== |
+ | {| | ||
+ | |[http://www.slightlymagic.net/forum/download/file.php?id=23333 LHpi.external_dependencies-v2.zip] | ||
+ | |- | ||
+ | |version||2 | ||
+ | |- | ||
+ | |updated last on||2015-10-08 | ||
+ | |- | ||
+ | |size||939.35 KiB | ||
+ | |} | ||
+ | {| | ||
+ | |[http://www.slightlymagic.net/forum/download/file.php?id=22770 external dependencies.zip] | ||
+ | |- | ||
+ | |version||1 | ||
+ | |- | ||
+ | |updated last on||2015-07-21 | ||
+ | |- | ||
+ | |size||203.32 KiB | ||
+ | |} | ||
+ | This archive contains additional libraries that are needed by LHpi.mkm-helper.lua and LHpi.dummyMA. | ||
+ | |||
+ | Since 0.6, the dummy uses LuaSockets to implement ma.GetUrl. If you need them, here are the additional files. Included are LuaSockets files for http connections and the Lua interpreter that the LuaSockets dlls were built against. Extract the <code>ext</code> and <code>bin</code> folders into the <code>lib</code> folder of your LHpi setup (usually in <code>Magic Album\Prices</code>). | ||
+ | |||
+ | LHpi.mkm-helper uses multiple third party libraries to add support for OAuth,SSL,JSON, all taken from open source projects. Refer to [[#third party libraries]] for details. Note that the archive made available here contains only the files needed by LHpi, not the whole projects. | ||
+ | |||
+ | ===How-To update sitescripts=== | ||
+ | ====Update with new set information==== | ||
+ | [[How-To sitescriptTemplate#Updating sitescripts with new set information]] | ||
+ | ====How-To resolve failed cards==== | ||
+ | [[How-To sitescriptTemplate#resolving failed cards]] | ||
+ | |||
+ | ===developer's notes=== | ||
+ | ====Planned features==== | ||
in no particular order | in no particular order | ||
− | * Promo and special sets for <s>magicuniverseDE</s>,mtgmintcards, tcgplayerPriceguide | + | * <s>Promo and special sets for <s>magicuniverseDE</s>,mtgmintcards, tcgplayerPriceguide</s> |
− | ** add all <s>special and</s> promo sets (cardcount,variants,foiltweak) to LHpi.Data | + | ** <s>add all <s>special and</s> promo sets (cardcount,variants,foiltweak) to LHpi.Data</s> |
* tweak log : better differentiation of log levels (normal,verbose,debug); rewrite normal-level logentries to be more pleasing to read.<br />I'd love to get users' feedback on this: how silent should normal level be; what should not be logged unless in debug mode? | * tweak log : better differentiation of log levels (normal,verbose,debug); rewrite normal-level logentries to be more pleasing to read.<br />I'd love to get users' feedback on this: how silent should normal level be; what should not be logged unless in debug mode? | ||
*<code>(((a+b)/2)+c)/2 != (a+b+c)/3 = (((a+b)/2*2)+c)/3</code><br/> For mathematically correct averaging, need to attach a counter to averaged prices, then on next averaging, do <code>if counter then newaverage=(oldaverage*(counter+1) + newprice) / (counter+2)</code> | *<code>(((a+b)/2)+c)/2 != (a+b+c)/3 = (((a+b)/2*2)+c)/3</code><br/> For mathematically correct averaging, need to attach a counter to averaged prices, then on next averaging, do <code>if counter then newaverage=(oldaverage*(counter+1) + newprice) / (counter+2)</code> | ||
+ | * generalize settweak from tcgplayerPriceGuide to library | ||
− | ===Possible features=== | + | ====Possible features==== |
in no particular order | in no particular order | ||
* passing importsets to BuildCardData could sort out the mess that is magicuniverse's promo page, at the cost of lots looping through sets. Should thus be done in site.BCDPlugins | * passing importsets to BuildCardData could sort out the mess that is magicuniverse's promo page, at the cost of lots looping through sets. Should thus be done in site.BCDPlugins | ||
Line 188: | Line 354: | ||
add feature requests here, please. | add feature requests here, please. | ||
− | ===Problems=== | + | ====Problems==== |
− | * | + | * number of cards in set must be hardcoded into library. An <code>ma.GetCardsInSet( setid,langid,regular|tokens )</code>[http://www.slightlymagic.net/forum/tracker.php?p=2&t=303] would help, but the numbers are easy to hardcode anyways. |
− | + | * cannot use mtgmintcard to get € prices. see [http://www.slightlymagic.net/forum/viewtopic.php?f=32&t=2324&start=285#p100271|here] for details. | |
− | + | * cannot automatically determine the currently running script's name. Neither <code>arg[0]</code> nor <code>debug.getinfo(function,'S')</code> are available in the embedded Lua. | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | </ | + | |
− | + | ||
− | * | + | |
− | * | + | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
===version history=== | ===version history=== | ||
Line 217: | Line 366: | ||
<br />In case you need an old version or just want to see how the scripts have evolved: The version numbers below are also download links. | <br />In case you need an old version or just want to see how the scripts have evolved: The version numbers below are also download links. | ||
− | ====pre-2.0==== | + | =====pre-2.0===== |
;1.2 (Downloaded 22 times) | ;1.2 (Downloaded 22 times) | ||
:first public release | :first public release | ||
Line 343: | Line 492: | ||
:* removed legacy conversion of DEBUGSKIPFOUND to DEBUGFOUND | :* removed legacy conversion of DEBUGSKIPFOUND to DEBUGFOUND | ||
:* call site.SetExpected if CHECKEXPECTED | :* call site.SetExpected if CHECKEXPECTED | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=20398 2.13] | ||
+ | :DoImport | ||
+ | :* finetune site.expected[setid].pset defaults according to site.expected.EXPECTTOKENS,site.expected.EXPECTNONTRAD and site.expected.EXPECTREPL | ||
+ | :* increased default dataver from 2 to 5 | ||
+ | :BuildCardData | ||
+ | :* set card.object type | ||
+ | :FillCardsetTable | ||
+ | :* also fill with objtype | ||
+ | :MergeCardrows | ||
+ | :* merge objtypes | ||
+ | :SetPrice | ||
+ | :* look for special variant names "token", "nontrad", "replica", "insert" and transform them into object types for ma.SetPrice | ||
+ | ;2.14 | ||
+ | :cardcount.nontr renamed to cardcount.nontrad | ||
+ | :GetSourceData split into GetSourceData and ParseSourceData. | ||
+ | :* this change is completely transparent to existing sitescripts, but having GetSourceData seperated allows to use it from the sitescript for special cases. | ||
+ | :LHpi.GetSourceData( sourceurl,urldetails ) | ||
+ | :* defer fetching to sitescript for oauth | ||
+ | :* now returns #string sourcedata | ||
+ | :* converts url to filename if OFFLINE | ||
+ | :* fixed savepath==nil handling (needed when called before DoImport has run) | ||
+ | :* Log https status if not ok | ||
+ | :LHpi.ParseSourceData( sourcedata,sourceurl,urldetails ) | ||
+ | :* does the parsing previously done by GetSourceData and returns the #table sourceTable | ||
+ | :* shortcut return nil for sourcedata==nil | ||
+ | :* now strictly expect site.ParseHtmlData to return card.price as #number (and not #table, as was possible before) | ||
+ | :* do not sanitize nor check card.regprice or card.foilpriceBuildCardData | ||
+ | :* counts namereplace and foiltweak even if card is dropped | ||
+ | :BuildCardData | ||
+ | :* remove "(nontrad)","(replica)","(plane)","(scheme)","(conspiracy)" and | ||
+ | :"(oversized)" card.name infixes when no longer needed | ||
+ | :* set objecttype replica for 105,106,69 | ||
+ | :* improved Token suffix handling | ||
+ | :* keeps "Replica" suffix for variants | ||
+ | :* removes "Insert" suffix | ||
+ | :* keep (oversized) in set [40] | ||
+ | :SetPrice | ||
+ | :* objtype infixe handling improved | ||
+ | :Logtable | ||
+ | :* no longer strictly requires #table arguments | ||
+ | :MainImportCycle | ||
+ | :* fixed CHECKEXPECTED handling for semi-available languages | ||
+ | :DoImport | ||
+ | :* changed site.expected.EXPECTTOKENS to site.expected.tokens | ||
+ | :* changed site.expected.EXPECTNONTRAD to site.expected.nontrad | ||
+ | :* changed site.expected.EXPECTREPL to site.expected.replica | ||
+ | :* all three can be #boolean (like before) or #table { [langid]=#boolean,... } | ||
+ | :* DoImport passes supImportfoil,supImportlangs,supImportsets to site.SetExpected | ||
+ | :* fixed expected defaults and checks | ||
+ | :removed some leftover commented-out code | ||
+ | :improved some comments and log msgs | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=22737 2.15] | ||
+ | :Split Initialize() from DoImport(importfoil , importlangs , importsets) | ||
+ | :always run Initialize() before returning LHpi library table. | ||
+ | :scriptname,savepath,logfile change from global to local (in site and LHpi) | ||
+ | :Log now filters out VERBOSE and DEBUG lines by loglevel param to save "if DEBUG then Log() end" conditionals | ||
+ | :*those conditionals have been cut and loglevel has been set for all existing LHpi.Log calls | ||
+ | :DEBUGVARIANTS=true no longer sets DEBUG=false, instead remembers and restores previous DEBUG state | ||
+ | :new LHpi.OAuthEncode(s) moved here from mkm-helper | ||
+ | :workdir fixes | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23328 2.16] | ||
+ | :removed leftover unused code | ||
+ | :removed deprecated parameter offline from site.BuildUrl calls | ||
+ | :site.resultregex findings (if any) logged in MainImportCycle instead of GetSourceData | ||
+ | :no longer check for Data in deprecated location | ||
− | ====LHpi.Data==== | + | =====LHpi.Data===== |
;[http://www.slightlymagic.net/forum/download/file.php?id=15850 1] | ;[http://www.slightlymagic.net/forum/download/file.php?id=15850 1] | ||
:initial release, split from LHpi library 2.6 | :initial release, split from LHpi library 2.6 | ||
Line 364: | Line 578: | ||
:fixed: 320,490,590,139,787,26,31,30,10,740,390 | :fixed: 320,490,590,139,787,26,31,30,10,740,390 | ||
:added lots of TODO markers | :added lots of TODO markers | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=20397 5] | ||
+ | :renamed Data.sets[setid].cardcount.overs to Data.sets[setid].cardcount.repl | ||
+ | :add "Token" suffix to variant tables to enable objtype=2 in LHpi.BuildCardData | ||
+ | :add Nontrad,Replica variants to 807,769,787,70,761,786,784,782,801,778,23,21,20 | ||
+ | :new sets 809,810,811,812,813 | ||
+ | :fix cardcount, variant and/or foiltweak tables for 807,812,31,30,25,23,21,10 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=22736 6] | ||
+ | :new 52,53,55,814,815,816,817,818,819,820,821,822 | ||
+ | :added lang [17]="Phyrexian" | ||
+ | :fixed names for 21,40,814 | ||
+ | :variant table for 15,22,25,26,27,33,40,45 | ||
+ | :foiltweak table for 22,26,33 | ||
+ | :cardcount fixed/updated in 9,105,106,762,791,800,813,816,30,22,21,26 | ||
+ | :fixed/updated variants for 43,69,201[ITA],758,764,766,781,787,800,813,22 | ||
+ | :commented variants in 180,140,110,100,90,280,230,220,170,260,105,25 | ||
+ | :foiltweak fixed/updated in 813 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23253 7] | ||
+ | :new 825,823,824,826 | ||
+ | :fixed variants for 822,819,25 | ||
+ | :updated cardcount for 822,22 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23329 8] | ||
+ | :variant table for 635 (Salvat Magic Encyclopedia; I'm glad that's finally done) | ||
+ | :add tla (three letter abbreviations) to all sets | ||
+ | :updated/fixed variants for 52,25,10 | ||
+ | :updated/fixed cardcount for 791,53,52,50,21,10 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23389 9] | ||
+ | :vupdated/fixed cardcount for 26 | ||
− | ====magicuniverseDE==== | + | =====LHpi.dummyMA===== |
+ | ;0 | ||
+ | :initial release as part of LHpi-v2.0.zip | ||
+ | [http://www.slightlymagic.net/forum/download/file.php?id=10165 0.1] | ||
+ | :moved to seperate file instead of including it in LHpi lib zip | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=15848 0.2] | ||
+ | :Backwards compatible with Lua5.1 | ||
+ | :tested with seperated LHpi.Data | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=16272 0.3] | ||
+ | :misc. small improvements to code and/or comments | ||
+ | :patch paths, because dev src has been moved from MA\Prices | ||
+ | :dummy.fakesitescript() | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=16292 0.4] | ||
+ | :execution timer in main() | ||
+ | :global variables changed to dummy.* | ||
+ | :new dummy.performancetest(repeats,script,impF,impL,impS,timefile) | ||
+ | :more comfort, less code duplication | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=20395 0.5] | ||
+ | :optional object type parameter in ma.SetPrice | ||
+ | :new sets 801,803,804,805,806,807,808,809,810,811,812,813 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=22734 0.6] | ||
+ | :new dummy.CompareDummySets(mapath,libver) | ||
+ | :new dummy.CompareDataSets(mapath,libver,dataver) | ||
+ | :new dummy.CompareSiteSets (moved from LHpi.magickartenmarkt.lua) | ||
+ | :new dummy.ListUnknownUrls(expansions,missing,file) (moved from LHpi.mkm-helper.lua) | ||
+ | :error handling in ma.GetFile and ma.Putfile | ||
+ | :for libver > 2.14 and Lua 5.2, use dofile instead of homemade loader | ||
+ | :CompareDataSets compares with dummy's set tables instead of Database/Sets.txt | ||
+ | :use global workdir as introduced in 2.15 | ||
+ | :added 52,53,55,814,815,816,817,818,819,820,821 | ||
+ | :fixed dummy.forceEnv | ||
+ | :fixed savepath handling | ||
+ | :deprecated dummy.loadlibonly and dummy.loadscript | ||
+ | :dummy.mergetables no longer changes param #table teins | ||
+ | :update helper functions ouput to log instead of stdout | ||
+ | :ma.GetUrl now uses luasockets instead of returning nil | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23332 0.7] | ||
+ | :can run in helper mode, doing nothing but providing ma.* namespace | ||
+ | :renamed from dummyMA.lua to LHpi.dummyMA.lua | ||
+ | :added LHpi.magickartenmarkt.lua and LHpi.mkm-helper.lua to script selection | ||
+ | :ma.PutFile and ma.GetFile only print params if DEBUG | ||
+ | :added 825,826,824,823 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23390 0.8] | ||
+ | :improved LE BOM handling | ||
+ | :function names CamelCased | ||
+ | |||
+ | =====external dependencies===== | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=22770 1] | ||
+ | :initial release for dummyMA-v0.6 | ||
+ | :LuaSockets to implement ma.GetUrl | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23333 2] | ||
+ | :release for LHpi.magickartenmarkt-v2.16.8.2 | ||
+ | :add support for OAuth,SSL,JSON | ||
+ | |||
+ | =====LHpi.sitescriptTemplate===== | ||
+ | ;2.0 | ||
+ | :initial release as part of LHpi-v2.0.zip | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=10170 2.1] | ||
+ | :use LHpi-v2.1 | ||
+ | :let LHpi lib set savepath | ||
+ | :moved to seperate file instead of including it in LHpi lib zip | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=10424 2.2.1] | ||
+ | :use LHpi-v2.2 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=13302 2.5.1] | ||
+ | :use LHpi-v2.5 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=15815 2.6.1] | ||
+ | :use LHpi-v2.6 | ||
+ | :pre-populated site.sets with all sets | ||
+ | :added LOGFOILTWEAK option | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=15857 2.7.1.1] | ||
+ | :use LHpi-v2.7 and LHpi.Data-v1 | ||
+ | :load lib from \lib subdir | ||
+ | :cleaned up luadoc and improved documentation | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=16129 2.8.1.1] | ||
+ | :uses LHpi-v2.8 and LHpi-Data-v1 | ||
+ | :site.frucs: new format | ||
+ | :sample BuildUrl adapted to new fruc format | ||
+ | :site.BCDplugins: added importfoil, importlangs parameters | ||
+ | :site.ParseHtmlData: now returns a container, to allow multiple cards | ||
+ | :new option STRICTCHECKEXPECTED | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=16273 2.9.2.8] | ||
+ | :too much simplification prevented possible functionality: reintroduced site.langs[langid].id field for reverse lookup | ||
+ | :split script version from scriptname to ease possible future change to versioning scheme | ||
+ | :misc. small improvements to example code and/or comments | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=16293 2.10.2.9] | ||
+ | :lib 2.10 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=18139 2.12.4.10] | ||
+ | :libver 2.12 | ||
+ | :dataver 4 | ||
+ | :BuildUrl: added optional fields to urldetails luadoc | ||
+ | :new (optional) field: site.pagenumberregex | ||
+ | :wrap site.expected in site.SetExpected() | ||
+ | :add missing sets | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=20396 2.13.5.11] | ||
+ | :new EXPECTNONTRAD and EXPECTREPL options to site.expected | ||
+ | :new sets 809,810,811,812,813 | ||
+ | :new STRICTOBJTYPE option | ||
+ | ;2.14.5.12 | ||
+ | :removed url to filename changes that are done by the library if OFFLINE | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=22735 2.15.6.13] | ||
+ | :script configuration variables local instead of global | ||
+ | :workdir support | ||
+ | :site.LoadLib split from ImportPrice | ||
+ | :site.Initialize loads LHpi lib if not yet available (needed if ImportPrice is not called) | ||
+ | :keep LHpi library table if already present | ||
+ | :no longer check for lib and Data in deprecated location | ||
+ | :removed site.CompareSiteSets | ||
+ | :optional site.BuildUrl("list") behaviour | ||
+ | :new optional function site.FetchExpansionList | ||
+ | :site.BuildUrl supports multiple urls per set | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23334 2.16.8.14] | ||
+ | :merged mkm branch | ||
+ | :fixed some comments | ||
+ | :added 825,823,824,826 | ||
+ | :defaults for site.expected[sid].namereplaced and site.expected[sid].foiltweaked | ||
+ | |||
+ | ====LHpi.magickartenmarkt==== | ||
+ | ;2.16.8.1 | ||
+ | :unreleased | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23330 2.16.8.2] | ||
+ | :initial release | ||
+ | :split mkm-helper | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23392 2.16.9.3] | ||
+ | :updated expected for 30,26 | ||
+ | :site.priceTypes format changed, descriptions added | ||
+ | :on HTTP errors, site.FetchSourceDataFromOAuth prints/logs errormsg and returns nil,status | ||
+ | :mkm price type choice in site.* namespace | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23445 2.16.9.4] | ||
+ | :Properly initialise requestCounter file if none exist (thanks to Danatar for reporting and testing) | ||
+ | |||
+ | =====LHpi.mkm-helper===== | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23331 2.16.8.1] | ||
+ | :initial release | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23391 2.16.9.2] | ||
+ | :GetSourceData gracefully skips set on http errors | ||
+ | :#table dataStaleAge { "default"=#number, #number setid=number, ... } instead of #number DATASTALEAGE | ||
+ | :new mode.forcerefresh | ||
+ | :new mode.checkstock (experimental, unpolished and largely undocumented) | ||
+ | :* new helper.CheckStockPrices(maStock, mkmStock) | ||
+ | :* new helper.GetStock() | ||
+ | :* new helper.LoadCSV( filename ) | ||
+ | :check dummy.version | ||
+ | :function names CamelCased | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23444 2.16.9.3] | ||
+ | :don't update dataAge if set was aborted (i.e. 429) | ||
+ | :changed mode.checksales to mode.checkstock (now matching the documentation) | ||
+ | :Properly initialise timestamp file if none exist (thanks to Danatar for reporting and testing) | ||
+ | |||
+ | ====LHpi.tcgplayerPriceGuide==== | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=9267 2.0] | ||
+ | :initial release | ||
+ | :just adapted Golob's avsets to LHpi syntax and tested basic functionality | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=10168 2.1] | ||
+ | :updated for MA v1.5.2.264 | ||
+ | :use LHpi-v2.1 | ||
+ | :let LHpi lib set savepath | ||
+ | :tested all core and expansions up to gatecrash, also 3 portal sets and both "Un-sets" | ||
+ | :added namereplace table and expected card counts | ||
+ | :commented out remaining untested sets | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=10425 2.2.1] | ||
+ | :Updated for MA 1.5.2.264b (includes Dragon's Maze) | ||
+ | :use LHpi-v2.2 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=11955 2.3.1] | ||
+ | :use LHpi-v2.3 | ||
+ | :added MMA and M14 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=13129 2.4.1] | ||
+ | :added Theros | ||
+ | :lots of Ae -> Æ replacements | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=13218 2.5.1] | ||
+ | :changed BuildUrl to match lib 2.5 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=15806 2.6.1] | ||
+ | :use LHpi-v2.6 | ||
+ | :added BNG,FTV:20,DD:HvsM,DD:SvT,DD:VvK,DD:EvT,DD:DvD,Beatdown | ||
+ | :now with less expected fails and slightly better support for card variants | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=15859 2.7.1.1] | ||
+ | :use LHpi-v2.7 and LHpi.Data-v1 | ||
+ | :load lib from \lib subdir | ||
+ | :cleaned site.sets' frucs | ||
+ | :added an option to just copy English card's prices to other languages | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=16208 2.8.1.1] | ||
+ | :use LHpi-v2.8 | ||
+ | :shortened site.langs | ||
+ | :simplified site.frucs,site.sets to single fruc | ||
+ | :hotfix: set options to default values | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=16279 2.9.2.10] | ||
+ | :synchronized with template | ||
+ | :drop all "SOON" prices (higher maintenance for site.expected,but less "0"s in averaging) | ||
+ | :updated site.expected (apparently, they don't like Boros: a sizable portion is Plains and Mountains!?) | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=17123 2.11.3.11] | ||
+ | :JOU | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=18137 2.12.4.12] | ||
+ | :added 808, most special and promo sets | ||
+ | :TODO-marked all others | ||
+ | :updated expected and namereplace tables | ||
+ | :synchronized with template | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=20402 2.13.5.13] | ||
+ | :new sets 810,812,813 | ||
+ | :removed oversized handling from BCDPluginPre | ||
+ | :fix 100,170,801,787,600,380,10,23 | ||
+ | :add "Token" suffix to replacement names | ||
+ | :update site.expected | ||
+ | :synchronized with template | ||
+ | ;2.13.5.14 | ||
+ | :added 814 | ||
+ | ;2.14.5.14 | ||
+ | :removed url to filename changes that are done by the library if OFFLINE | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=22742 2.15.6.14] | ||
+ | :added 816,818,815,817,819,820,821,40,32,27,25,23,22,21,822,340,55,45,822 | ||
+ | :new features from template/mkm branch | ||
+ | :site.BuildUrl | ||
+ | :* supports multiple urls per set | ||
+ | :* set url.details.setid | ||
+ | :special handling for | ||
+ | :* Dragonfury cards | ||
+ | :* unusual lang cards | ||
+ | :updated/fixed/added variants,namereplacements,expected | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23252 2.15.7.15] | ||
+ | :added 825,823,824,826 | ||
+ | :fixed namereplace 822,808,110,100,90,751,570,220,170,820,819,814,807,801,796,787,600,45,23,22 | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=23327 2.16.8.16] | ||
+ | :synced with template | ||
+ | :updated site.namereplace for 825,220,170,820,40 | ||
+ | :updated site.expected | ||
+ | |||
+ | ====LHpi.magicuniverseDE==== | ||
;[http://www.slightlymagic.net/forum/download/file.php?id=9264 2.0] | ;[http://www.slightlymagic.net/forum/download/file.php?id=9264 2.0] | ||
:initial release | :initial release | ||
Line 401: | Line 866: | ||
:fixed site.langs | :fixed site.langs | ||
:synchronized with template | :synchronized with template | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=20399 2.13.5.13] | ||
+ | :new sets 807,813 | ||
+ | :removed oversized handling from BCDPluginPre | ||
+ | :add "Token" suffix to replacement names | ||
+ | :update site.expected | ||
+ | :fix 140,420,801 | ||
+ | :synchronized with template | ||
+ | ;2.13.5.14 | ||
+ | :addded 813 | ||
+ | ;2.14.5.14 | ||
+ | :removed url to filename changes that are done by the library if OFFLINE | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=22775 2.15.6.14] | ||
+ | synchronized with template | ||
+ | fixed/updated site.regex | ||
− | ====mtgmintcard==== | + | ====LHpi.mtgmintcard==== |
;[http://www.slightlymagic.net/forum/download/file.php?id=9265 2.0] | ;[http://www.slightlymagic.net/forum/download/file.php?id=9265 2.0] | ||
:initial release | :initial release | ||
Line 443: | Line 922: | ||
:updated all expected counts | :updated all expected counts | ||
:synchronized with template | :synchronized with template | ||
− | + | ;[http://www.slightlymagic.net/forum/download/file.php?id=20400 2.13.5.13] | |
− | + | :new sets 809,810,812,813 | |
− | ;[http://www.slightlymagic.net/forum/download/file.php?id= | + | :removed oversized handling from BCDPluginPre |
− | + | :add "Token" suffix to replacement names | |
− | + | :update site.expected | |
− | + | :fix 330,801,787,600 | |
− | : | + | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | : | + | |
− | : | + | |
− | + | ||
− | + | ||
− | + | ||
− | : | + | |
− | : | + | |
− | + | ||
− | + | ||
:synchronized with template | :synchronized with template | ||
− | + | ;2.13.6.14 | |
− | : | + | :added 814 |
− | ; | + | ;2.14.5.14 |
− | : | + | :removed url to filename changes that are done by the library if OFFLINE |
− | ;[http://www.slightlymagic.net/forum/download/file.php?id= | + | ;[http://www.slightlymagic.net/forum/download/file.php?id=22773 2.15.6.14] |
− | + | ||
− | + | ||
− | + | ||
:synchronized with template | :synchronized with template | ||
− | ====trader-onlineDE==== | + | ====LHpi.trader-onlineDE==== |
;2.0 | ;2.0 | ||
:unreleased | :unreleased | ||
Line 527: | Line 971: | ||
:added 801,807,808 | :added 801,807,808 | ||
:updated expected 100,680,690,800 | :updated expected 100,680,690,800 | ||
+ | :synchronized with template | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=20403 2.13.5.13] | ||
+ | :new sets 808,812,813 | ||
+ | :add "Token" suffix to replacement names | ||
+ | :update site.expected | ||
+ | :fix 784,801,600 | ||
+ | :synchronized with template | ||
+ | ;2.13.6.13 | ||
+ | :added 814 | ||
+ | ;2.14.5.13 | ||
+ | :removed url to filename changes that are done by the library if OFFLINE | ||
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=22772 2.15.6.14] | ||
:synchronized with template | :synchronized with template | ||
− | ====mtgpprice.com==== | + | ====LHpi.mtgpprice.com==== |
+ | ;[http://www.slightlymagic.net/forum/download/file.php?id=16280 2.9.2.1] | ||
+ | :initial release, unfinished | ||
;[http://www.slightlymagic.net/forum/download/file.php?id=17606 2.11.3.3] | ;[http://www.slightlymagic.net/forum/download/file.php?id=17606 2.11.3.3] | ||
:first semi-official release | :first semi-official release | ||
+ | :still largely unmaintained | ||
+ | :JOU,DDM | ||
;[http://www.slightlymagic.net/forum/download/file.php?id=18138 2.12.4.3] | ;[http://www.slightlymagic.net/forum/download/file.php?id=18138 2.12.4.3] | ||
:site.regex,ParseHtmlData fixed for new site layout | :site.regex,ParseHtmlData fixed for new site layout | ||
Line 538: | Line 998: | ||
:some more of namereplacement for basic land variants | :some more of namereplacement for basic land variants | ||
:fixed offline urls | :fixed offline urls | ||
− | + | ;[http://www.slightlymagic.net/forum/download/file.php?id=20401 2.13.5.4] | |
− | + | :new set 813 | |
− | + | :fix 250,180 (except basic lands) | |
− | + | :synchronized with template | |
− | ;[http://www.slightlymagic.net/forum/download/file.php?id= | + | ;2.13.6.5 |
− | + | :added 814 | |
− | + | ;2.14.5.5 | |
− | + | :removed url to filename changes that are done by the library if OFFLINE | |
− | + | ;[http://www.slightlymagic.net/forum/download/file.php?id=22774 2.15.6.5] | |
− | + | :synchronized with template | |
− | + | ||
− | : | + | |
− | + | ||
− | : | + | |
− | + | ||
− | + | ||
− | ; | + | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | ; | + | |
− | : | + | |
− | + | ||
− | + | ||
− | ;[http://www.slightlymagic.net/forum/download/file.php?id= | + | |
− | : | + | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
==License== | ==License== | ||
− | The LHpi library and all its sitescripts are released as open source under the | + | ====LHpi==== |
+ | The LHpi library and all its sitescripts are released as open source under the [http://www.gnu.org/licenses/gpl.html GNU General Public License]. Please note that LHpi is thusly licensed seperately from Magic Album proper, which to date is closed source. | ||
+ | ====third party libraries==== | ||
+ | LHpi.dummyMA, LHpi.magickartenmarkt and LHpi.mkm-helper also require external libraries. Those are taken from the following projects: | ||
+ | =====Lua===== | ||
+ | lua52.exe and lua52.dll are distributed under the [http://www.lua.org/license.html MIT license]. | ||
+ | =====LuaSocket===== | ||
+ | [http://w3.impa.br/~diego/software/luasocket/ LuaSocket] is © 2004-2007 Diego Nehab and distributed under the same terms and conditions as the Lua language. | ||
+ | =====dkjson===== | ||
+ | [http://dkolf.de/src/dkjson-lua.fsl/home dkjson] is © 2010-2013 David Heiko Kolf and distributed under the same conditions as the Lua interpreter. | ||
+ | =====LuaOAuth===== | ||
+ | [https://github.com/ignacio/LuaOAuth LuaOAuth] is © 2010 Ignacio Burgueño and distributed under the [https://github.com/ignacio/LuaOAuth/blob/master/LICENSE MIT License]. It contains code from lua-multipart-post © 2012-2013 by Moodstocks SAS under the same license. | ||
+ | =====base64.lua===== | ||
+ | [http://lua-users.org/wiki/BaseSixtyFour base64.lua] is © 2009 by Alex Kloss, licensed under the terms of the [http://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html LGPL2]. | ||
+ | =====LuaCrypto===== | ||
+ | [http://luacrypto.luaforge.net/ LuaCrypto] is © 2006 Keith Howe and uses the same [http://luacrypto.luaforge.net/license.html license] as Lua 5.0. | ||
+ | =====LuaSec===== | ||
+ | [https://github.com/brunoos/luasec LuaSec 0.5] is © 2006-2014 Bruno Silvestre and distributed under the [https://github.com/brunoos/luasec/blob/master/LICENSE MIT License] | ||
+ | =====OpenSSL===== | ||
+ | [https://www.openssl.org/ OpenSSL] © 1998-2011 The OpenSSL Project, licensed under [https://www.openssl.org/source/license.html OpenSSL License]. This product includes cryptographic software written by Eric Young (eay@cryptsoft.com) and software written by Tim Hudson (tjh@cryptsoft.com), licensed under the [https://www.openssl.org/source/license.html Original SSLeay License]. | ||
+ | ====Cryptograpy export notice==== | ||
+ | Import and/or use of strong cryptography software is illegal in some parts of the world. If this applies to you do not download the external dependecies file. By downloading this software, you agree to and take full responsibility to abide by the relevant laws. | ||
==Other scripts== | ==Other scripts== |
Latest revision as of 10:48, 19 October 2015
LHpi is LordHelmchen's price import library and sitescripts for Magic Album.
LHpi is a collection of lua scripts to be used by Magic Album, but is not part of Magic Album and thus updated and supported separately.
Discuss, ask for help and leave feedback at the corresponding Forum Thread.
Contents
|
[edit] Sitescripts
The sitescripts are the actual price import scripts, written in lua. They use the Magic Album Lua API and #the LHpi Library to query and parse a website and import the price information into Magic Album. Most of the program logic is contained in the library, so the sitescript needs to hold only site-specific information and a few functions that are not easily generalizable.
In theory, you should not be dependent on a new version once a new set is available: just add new entries to site.sets table in the sitescript. Being easy to update yourself was one of the design goals.
Sitescripts will need to provide
- a table of supported sets with language and URL infix fields
- a table of supported languages with URL infix field
- a function to build URL
- a function to parse the HTML raw data into data fields via regular expressions (and the regex itself)
- optionally, they can also provide
- functions for site specific data manipulation, in case the site has some strange formatting the library does not understand
- a one-to-one name map for names that differ between the site and Magic Album's database
- tables to map card names to picture versions, in case the one-to-one map and LHpi's defaults are not sufficient
- a table with expected card count numbers for LHpi's sanity checks.
- a function to fetch a list of expansions from the site, to be used by update helper functions.
[edit] magickartenmarkt
LHpi.magickartenmarkt.lua.zip | |
Sitescript version | 2.16.9.4 |
updated last on | 2015-10-19 |
size | 42.04 KiB |
MagicKartenMarkt (or magiccardmarket is "Europe's largest online marketplace for Magic the Gathering cards!". It has become the de facto standard way to acquire singles, at least in my local MtG circles.
They are the only price source that I personally am interested in anymore, but the script is not in a releasable state yet. See the forum thread, starting at state of things for details on how far along development is. Due to the limitations imposed by Magic Album's embedded Lua interpreter and the MKM API, you cannot fetch live data with the sitescript. You can use LHpi.mkm-helper.lua to download price data.
Additional preparation steps are needed for LHpi.magickartenmarkt. Please refer to the #additional instructions for MKM below.
[edit] mkm-helper.lua
LHpi.mkm-helper.zip | |
Sitescript version | 2.16.9.3 |
recommended dummyMA version | >=0.8 |
updated last on | 2015-10-19 |
size | 10.78 KiB |
Helper script to download price data from MKM. The data can then be imported to MA with magickartenmarkt.lua.
The MKM API imposes a limit of 5000 request per day. With over 30000 English cards known to Magic Album, mkm-helper tries to help space out the download over several days and keeps a record of when a set was last downloaded.
Also included is a batch file (LHpi.mkm-helper.bat) for ease of use. Edit it to point to your MA\Prices directory and provide a list of sets you want downloaded.
[edit] tcgplayer.com PriceGuide
LHpi.tcgplayerPriceGuide.lua.zip | |
Sitescript version | 2.16.8.16 |
updated last on | 2015-10-08 |
size | 28.24 KiB |
Tcgplayer Price Guide claims to accumulate prices from 75 online vendors. They are priced in $.
As requested by Golob here.
Their tables give a high, low and average price; the sitescript contains a variable to select which one of those it should use.
To change the column to import from, open the script in a text editor, find himelo = 3
and change it to himelo = 1
for high or himelo = 2
for medium prices. You can have multiple copies of the same script
Note that cards with multiple versions are averaged in some sets and set to "variant 1"'s price in others, as the source list does not clearly identify the version. Even when it does, it is unclear if the entry without version suffix is meant to be all unspecified or version 1 only. In most cases, "CARD" price is set for all versions and averaged with "CARD (version)" price for given versions now. Lately, their version handling has somewhat improved, but there's still a lot of guesswork involved...
[edit] magicuniverse.de
LHpi.magicuniverseDE.lua.zip | |
Sitescript version | 2.15.6.14 |
updated last on | 2015-07-21 |
size | 15.57 KiB |
Magic Universe is a German site with German and English cards, priced in €.
This site started it all, since I wanted prices in local currency, and my FLG apparently used them as price reference.
MAINTAINER WANTED! I do not need any prices besides from MKM, and will not update this (unless specifically requested) anymore. The last test results were:
:-( persetcount for Magic Origins (id 822) differs from expected. :-( counted : 267 set & 5 failed English cards ; 267 set & 5 failed German cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. :-( expected: 286 set & 0 failed English cards ; 286 set & 0 failed German cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. Total counted : 19607 set, 457 failed English cards 17544 set, 492 failed German cards 1 set, 0 failed French cards 19 set, 0 failed Italian cards ; 1102 dropped, 190 namereplaced and 19 foiltweaked. Total expected: 19630 set, 442 failed English cards 17926 set, 480 failed German cards 1 set, 0 failed French cards 19 set, 0 failed Italian cards ; 1132 dropped, 191 namereplaced and 15 foiltweaked. count differs in 27 sets: { [139]="Revised Edition (FBB)";[150]="Legends";[160]="The Dark";[180]="4th Edition";[190]="Ice Age";[220]="Alliances";[230]="Mirage";[280]="Tempest";[300]="Exodus";[330]="Urza’s Saga";[400]="Mercadian Masques";[410]="Nemesis";[420]="Prophecy";[430]="Invasion";[460]="7th Edition";[470]="Apocalypse";[480]="Odyssey";[520]="Onslaught";[560]="Mirrodin";[590]="Champions of Kamigawa";[670]="Coldsnap";[800]="Theros";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[822]="Magic Origins";}
[edit] mtgmintcard.com
LHpi.mtgmintcard.lua.zip | |
Sitescript version | 2.15.6.14 |
updated last on | 2015-07-21 |
size | 14.02 KiB |
Functionally the same as #"MTG Mint Card.lua" by Goblin Hero and Stromglad1.
When I wanted to generalize my script for easy adaption to other sites, this provided me with a good baseline, since I could search MA for cards that my script missed but should have set. Also, the site is easy to query and parse. They do have a currency chooser, so I started by trying to make a mtgmintcard-Euro script. I failed, but salvaged the wreck for default $ prices.
MAINTAINER WANTED! I do not need any prices besides from MKM, and will not update this (unless specifically requested) anymore. The last test results were:
:-( persetcount for Magic Origins (id 822) differs from expected. :-( counted : 264 set & 37 failed English cards ; 255 set & 46 failed Simplified Chinese cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. :-( expected: 286 set & 0 failed English cards ; 272 set & 0 failed Simplified Chinese cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. Total counted : 24130 set, 128 failed English cards 8217 set, 307 failed Simplified Chinese cards ; 850 dropped, 136 namereplaced and 46 foiltweaked. Total expected: 24352 set, 67 failed English cards 8780 set, 243 failed Simplified Chinese cards ; 813 dropped, 137 namereplaced and 39 foiltweaked. count differs in 22 sets: { [390]="Starter 1999";[570]="Darksteel";[630]="9th Edition";[700]="Planar Chaos";[740]="Duel Decks: Elves vs. Goblins";[755]="Duel Decks: Jace vs. Chandra";[757]="Duel Decks: Divine vs. Demonic";[758]="Alara Reborn";[762]="Zendikar";[763]="Duel Decks: Garruk vs. Liliana";[778]="Magic: The Gathering Commander";[784]="Dark Ascension";[786]="Avacyn Restored";[800]="Theros";[808]="Magic 2015";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[819]="Modern Masters 2015 Edition";[820]="Duel Decks: Elspeth vs. Kiora";[822]="Magic Origins";}
[edit] trader-online.de
LHpi.trader-onlineDE.lua.zip | |
Sitescript version | 2.15.6.14 |
updated last on | 2015-07-21 |
size | 16.57 KiB |
Trader Online is a German site with German and English cards, priced in €.
I found this site while looking for Token images, found it to be parsable by sitescript, and decided to use it as a test on how well the library was adapted to support additional sites.
MAINTAINER WANTED! I do not need any prices besides from MKM, and will not update this (unless specifically requested) anymore. The last test results were:
:-( persetcount for Magic Origins (id 822) differs from expected. :-( counted : 281 set & 5 failed English cards ; 270 set & 16 failed German cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. :-( expected: 286 set & 0 failed English cards ; 286 set & 0 failed German cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. Total counted : 22346 set, 962 failed English cards 19547 set, 117 failed German cards 429 set, 0 failed Italian cards ; 1314 dropped, 333 namereplaced and 11 foiltweaked. Total expected: 23069 set, 40 failed English cards 20536 set, 53 failed German cards 429 set, 0 failed Italian cards ; 1332 dropped, 339 namereplaced and 11 foiltweaked. count differs in 12 sets: { [100]="Beta";[105]="Collectors’ Edition (Domestic)";[106]="Collectors’ Edition (International)";[110]="Unlimited";[800]="Theros";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[819]="Modern Masters 2015 Edition";[820]="Duel Decks: Elspeth vs. Kiora";[822]="Magic Origins";}
[edit] mtgprice.com
LHpi.mtgprice.com.lua.zip | |
Sitescript version | 2.15.6.5 |
updated last on | 2015-07-21 |
size | 14.34 KiB |
I did not plan to include it in the sitescript collection, but now that the cat is out of the bag and it has been downloaded a dual-digit number of times, I can at least upload the newer version to make it compatible with the latest lib and data... Contains a variable to select between "Fair Trade" and "best Buy" price column. It still needs a lot of namereplace tables. I'm not that much interested in $ prices, so if you are, consider volunteering to finish and maintain it. Post in the thread if you want to do it but are unsure what to do.
MAINTAINER WANTED! I do not need any prices besides from MKM, and will not update this anymore (unless specifically requested). The last test results were:
:-( persetcount for Magic Origins (id 822) differs from expected. :-( counted : 225 set & 10 failed English cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. :-( expected: 272 set & 0 failed English cards ; 0 dropped, 0 namereplaced and 0 foiltweaked. Total counted : 25436 set, 610 failed English cards ; 0 dropped, 311 namereplaced and 150 foiltweaked. Total expected: 26027 set, 142 failed English cards ; 0 dropped, 380 namereplaced and 298 foiltweaked. count differs in 56 sets: { [5]="Book Inserts";[6]="Comic Inserts";[7]="Magazine Inserts";[9]="Video Game Promos";[10]="Junior Series Promos";[21]="Release & Launch Parties Promos";[22]="Prerelease Promos";[23]="Gateway & WPN Promos";[25]="Judge Gift Cards";[26]="Magic Game Day";[27]="Alternate Art Lands";[30]="Friday Night Magic Promos";[40]="Arena/Colosseo Leagues Promos";[42]="Summer of Magic Promos";[120]="Arabian Nights";[160]="The Dark";[190]="Ice Age";[200]="Chronicles";[201]="Renaissance";[225]="Introductory Two-Player Set";[260]="Portal";[270]="Weatherlight";[320]="Unglued";[380]="Portal Three Kingdoms";[390]="Starter 1999";[415]="Starter 2000";[450]="Planeshift";[490]="Deckmasters";[550]="8th Edition";[600]="Unhinged";[630]="9th Edition";[670]="Coldsnap";[680]="Time Spiral";[690]="Time Spiral Timeshifted";[756]="Conflux";[769]="Archenemy";[776]="New Phyrexia";[782]="Innistrad";[784]="Dark Ascension";[787]="Planechase 2012 Edition";[791]="Return to Ravnica";[793]="Gatecrash";[794]="Duel Decks: Sorin vs. Tibalt";[799]="Duel Decks: Heroes vs. Monsters";[800]="Theros";[801]="Commander 2013 Edition";[805]="Duel Decks: Jace vs. Vraska";[807]="Conspiracy";[809]="From the Vault: Annihilation";[812]="Duel Decks: Speed vs. Cunning";[813]="Khans of Tarkir";[814]="Commander 2014 Edition";[816]="Fate Reforged";[818]="Dragons of Tarkir";[820]="Duel Decks: Elspeth vs. Kiora";[822]="Magic Origins";}
[edit] the LHpi Library
LHpi-v2.16.lua.zip | |
Tested with Ma version | 1.6.4.348 |
recommended Data version | >=8 |
updated last on | 2015-10-08 |
size | 19.53 KiB |
LHpi-v2.15.lua.zip | |
Tested with Ma version | 1.6.4.348 |
recommended Data version | >=6 |
updated last on | 2015-07-18 |
size | 21 KiB |
It is not expected to be fast, but as reliable as possible:
It first compares the user requested sets and languages with what the sitescript has available and builds a table of urls to querry, seperated by set.
Then it loops though the sets that are both available and selected:
The raw html data is fetched, parsed and entered into an intermediate table of all prices for the current edition, merging regular and foil prices to minimize calls to ma.SetPrice.
Should the same card (defined by set,card,language,(picture-)version ) be encountered again, it compares the prices. If they are equal, the new price is discarded; if they differ, it uses the average price. Cards that are suffxed with a condition description are currently disregarded.
Once that cardsetTable is completed (all urls belonging to the set are parsed), it sends the prices to MA.
Finally, it compares the count of set (and failed to be set) prices with the number of cards in the set (or, if given, the expectation given in the sitescript).
See How-To_sitescriptTemplate#procedure_of_function_calls for details.
Logging to a seperate file is now disabled by default again (but still configurable by the variable SAVELOG in the script). The log (\Prices\LHpi.log) is overwritten on subsequent (re-)runs of any script.
[edit] LHpi.Data
LHpi.Data-v9.lua.zip | |
Library version | >=2.15 |
updated last on | 2015-10-15 |
size | 55.71 KiB |
LHpi.Data-v7.lua.zip | |
Library version | >=2.15 |
updated last on | 2015-09-28 |
size | 41.96 KiB |
Since library version 2.7, site-independent set data (number of cards,foilage, default variant and foiltweak tables) were moved into a seperate data file that gets loaded by the library on initialization. This will allow to distinguish between code updates (to the library) and set data updates (which are usually needed when a new set is released). Version numbering of the sitescripts will reflect both library and data versions used.
[edit] Installation and usage
- Download at least one sitescript and both #the LHpi Library and data file in the version the sitescript uses.
- note the change to version numbers and file names in 2.9.2 , see #version history for details.
- Unzip into the "Prices" subdirectory of your Magic Album installation.
- (
recommended, but not yet strictly necessary) Create a subfolder called "lib" under "Prices" and move both library and data file into the subfolder. - (optional) If you want to use the SAVEHTML or SAVETABLE feature, you need to create an empty subdirectory, named like the sitescript without file extension (for example "Prices\LHpi.magicuniverseDE").
- (re)start Magic Album and use the sitescript (not the library nor the data file. Step 3 makes this warning obsolete.) from Magic Album's Price Manager dialog.
[edit] additional instructions for MKM
- Please update #the LHpi Library to version >=2.16 and data to >=8, if you have not done so yet.
- Step 3 from the basic instructions is not optional anymore. Library components need to be in the Prices\lib\ directory.
- Download the LHpi.magickartenmarkt #sitescript and #helper and unzip into Prices\.
- Download #dummyMA (>=0.8) and #additional dependencies (>=2) and unzip into Prices\lib. Take care to keep the subdirectory structure of the dependencies intact.
- Obviously, we do need a place to store the downloaded price data, so as in step 4 above, create a subdirectory in Prices\. Unless you manually changed the scripts, it needs to be called "LHpi.magickartenmarkt"
- go to your MKM account page and create a set of app security tokens. Then create a copy of lib\LHpi.mkm.tokens.example and edit it with your personal MKM tokens. Do not share this information with anyone!
- LHpi only posts information retrieval requests through the MKM API, but I cannot accept any liability for anything that happens to your account.
- If you're unsure whether to trust my scripts, please read LHpi's sourcecode before making a decision. :-)
- Also, do note that the tokens are stored in a plaintext file on your computer (the one you create in this step) and, while LHpi.magickartenmarkt.lua or LHpi.mkm-helper are running, in memory.
- edit LHpi.magickartenmarkt.lua to change necessary settings:
- change
local mkmtokenfile = "LHpi.mkm.tokens.example"
to the token file you created in step 5. - MKM provides multiple price types. Choose two and map them to MA's regular and foil price columns by editing
local useAsRegprice=5
andlocal useAsFoilprice=6
(see details in the script's comments). - change
--OFFLINE = true
toOFFLINE = true
. I left it in so LHpi.magickartenmarkt can display an error and mention LHpi.mkm-helper, in case you did not read this instructions ;-)
- change
- (optional) edit LHpi.mkm-helper to set which sets should be downloaded by default (
MODE = { download=true, sets="standard" }
) and how long the data is valid (local dataStaleAge = { ["default"] = 60*60*24*7, -- one week, ... }
) - edit LHpi.mkm-helper.bat
- change
cd /D D:\Magic - The Gathering\Magic Album\Prices
to point to your MA location. - In addition or alternative to MODE in LHpi.mkm-helper.lua, you can set download mode and sets to be downloaded as arguments in LHpi.mkm-helper.bat; you might want to change the preconfigured example set definitions to your personal need.
- if you prefer commandline operation, you can also pass arguments to the batch file (as long as you keep the
%*
arguments intact when making changes). - MODE allows for one set definition, either a single string or a table. In the batch or as comand line argument, you can pass multiple set definitions. Valid strings are "core","expansion","special","promo", Magic Album set ids, set names or three letter codes.
- If you want, you can create multiple copies of the batch file for easy access to different modes of operation. Due to the included
cd
instruction, those can be stored anywhere on your computer.
- change
- Run LHpi.mkm-helper.bat to download pricing information. Then you can use LHpi.magickartenmarkt via Magic Album's Price Manager to import the downloaded data.
- At first, you will probably encounter HTTP 429 errors (too many requests). After each sucessfull download, the date and number of requests needed for the set will be recorded, and the errors will hopefully happen less frequent.
- To reset the request counter, pass "resetcounter" argument to the script (via commandline or by editing the batch file).
[edit] Troubleshooting
If something does not happen as expected
- make sure the script is saved utf-8 encoded. It's customary for lua scripts to be saved ANSI encoded, but as MA expects utf input, this is what I considered the most wysiwyg approach to set cards with special characters in their names.
- open the lua file in a text editor and change "--VERBOSE = true" to "VERBOSE = true", likewise enable "LOGDROPS","LOGNAMEREPLACE","LOGFOILTWEAK","STRICTEXPECTED" and "SAVELOG". This will make the script provide more feedback to help track down any problems (or just to see what it was doing) and also enable additional checks. The logfile will be called "[MA Folder]\Prices\[scriptname].log" by unless manually configured otherwise.
I'll try to answer any questions you might have. If the script misbehaves please follow the two steps above and attach a copy of the .log to your post.
[edit] Development
[edit] How-to write your sitescript file
Detailed instructions on the properties and functions to define in your sitescript file can be found at How-To sitescriptTemplate.
[edit] Hints for developers
Some information that might be useful if you want to write/improve sitescripts. Please add questions related to developing here, so I know what needs to be documented.
moved to How-To_sitescriptTemplate#Hints_for_developing
[edit] sitescriptTemplate
LHpi.sitescriptTemplate-v2.16.8.14.lua.zip | |
Sitescript version | 2.16.8.14 |
updated last on | 2015-10-08 |
size | 13.67 KiB |
This is just an empty, commented template to help write new sitescripts.
[edit] dummyMA
LHpi.dummyMA.zip | |
version | 0.8 |
updated last on | 2015-10-14 |
size | 11.02 KiB |
Magic Album API dummy to test and debug LHpi within an IDE (such as eclipse) and without needing Magic Album. LHpi.mkm-helper needs this for ma.GetFile, ma.PutFile, dummy.mergetables and a few tables (corests,expansionsets,specialsets,promosets).
[edit] additional dependencies
LHpi.external_dependencies-v2.zip | |
version | 2 |
updated last on | 2015-10-08 |
size | 939.35 KiB |
external dependencies.zip | |
version | 1 |
updated last on | 2015-07-21 |
size | 203.32 KiB |
This archive contains additional libraries that are needed by LHpi.mkm-helper.lua and LHpi.dummyMA.
Since 0.6, the dummy uses LuaSockets to implement ma.GetUrl. If you need them, here are the additional files. Included are LuaSockets files for http connections and the Lua interpreter that the LuaSockets dlls were built against. Extract the ext
and bin
folders into the lib
folder of your LHpi setup (usually in Magic Album\Prices
).
LHpi.mkm-helper uses multiple third party libraries to add support for OAuth,SSL,JSON, all taken from open source projects. Refer to #third party libraries for details. Note that the archive made available here contains only the files needed by LHpi, not the whole projects.
[edit] How-To update sitescripts
[edit] Update with new set information
How-To sitescriptTemplate#Updating sitescripts with new set information
[edit] How-To resolve failed cards
How-To sitescriptTemplate#resolving failed cards
[edit] developer's notes
[edit] Planned features
in no particular order
-
Promo and special sets for <s>magicuniverseDE,mtgmintcards, tcgplayerPriceguide</s>-
add all <s>special andpromo sets (cardcount,variants,foiltweak) to LHpi.Data</s>
-
- tweak log : better differentiation of log levels (normal,verbose,debug); rewrite normal-level logentries to be more pleasing to read.
I'd love to get users' feedback on this: how silent should normal level be; what should not be logged unless in debug mode? (((a+b)/2)+c)/2 != (a+b+c)/3 = (((a+b)/2*2)+c)/3
For mathematically correct averaging, need to attach a counter to averaged prices, then on next averaging, doif counter then newaverage=(oldaverage*(counter+1) + newprice) / (counter+2)
- generalize settweak from tcgplayerPriceGuide to library
[edit] Possible features
in no particular order
- passing importsets to BuildCardData could sort out the mess that is magicuniverse's promo page, at the cost of lots looping through sets. Should thus be done in site.BCDPlugins
- ideally, most information in LHpi.Data.sets could be read from MA via API getter functions.
- add hardcoded exchange rate for crude currency conversion between parsing and importing.
-
handle conflicting prices differently. Is averaging ok, or should the script discard one (which?) of the prices and/or throw an error ?Until further notice, I'm ok withhow averaging works.averaging conflicts. How it's done needs some work. - import prices from condition-suffixed cards instead of dropping them.
- patch to accept entries with a condition description if no other entry with better condition is in the table:
- buildCardData will need to add condition to carddata
- global conditions{} to define priorities
- then fillCardsetTable needs a new check before overwriting existing data
- patch to accept entries with a condition description if no other entry with better condition is in the table:
- add another sitescript (and adapt library if needed)
add feature requests here, please.
[edit] Problems
- number of cards in set must be hardcoded into library. An
ma.GetCardsInSet( setid,langid,regular|tokens )
[1] would help, but the numbers are easy to hardcode anyways. - cannot use mtgmintcard to get € prices. see [2] for details.
- cannot automatically determine the currently running script's name. Neither
arg[0]
nordebug.getinfo(function,'S')
are available in the embedded Lua.
[edit] version history
I decided to start all versions at 2.0, since it's the continuation of magicuniverseDEv1.6.lua, but with enough changes to warrant the next major version number. Do not expect revision numbers to stay synched. Starting with 2.2.1, sitescript minor version will conform to used library version, and the sitecripts will have their version suffixed.Starting with 2.7 sitescript versioning will be libversion.dataversion.sitescriptrevision. This should make it more clear which versions to keep and what can be deleted after updating. The sitescripts will know which library and data version they need and attempt to load the right one or exit gracefully.
Staring with 2.9, the versioning was changed yet again: libversion.dataversion.sitescriptrevision still stands, but the version number will no longer be suffixed to the sitescripts' filename. They still are kept inside the script. This means that the same .csv will be used by Magic album to store the imported prices through updates, and also makes drop-in updates of library and/or datafile simpler. Sitescript revision numbers have been raised to reflect the number of past releases.
In case you need an old version or just want to see how the scripts have evolved: The version numbers below are also download links.
[edit] pre-2.0
- 1.2 (Downloaded 22 times)
- first public release
- 1.3 (Downloaded 20 times)
- started externalizing site-specific data to seperate .config . exeptions for some sets still remain in the code.
- 1.4 (Downloaded 29 times)
- sanity checks should be sane again :)
- config externalization progressed far enough that adding new sets should be easy
- 1.5 (Downloaded 5 times)
- added RtR to avsets
- updated expected fail and drop counts
- continued config externalization and script generalization efforts
- moved gpl.txt from within the zip to seperate attachment
- 1.6 (Downloaded 1 time)
- last release under the name magicuniverseDEv1.x.lua
- added Gatecrash to avsets and expectedcounts (shop does not yet provide foil)
- renamed .config to .lconf to ease syntax highlighting
- got rid of utf-8 BOM
- changed some comments to luadoc-like
[edit] LHpi lib
- 2.0
- completely refactored, I'll call the externalization complete:
- config became sitescript, old script (and work-in-progress mtgmintcardEURO script) became the library.
- improved language handling
- slightly improved conflict handling: same price is merged, different price is averaged.
- 2.1
- added number of cards per set
- if no expectation is defined in sitescript, expect all cards and 0 tokens to be set successfully
- minor fix to debug loging
- moved handling of undefined sitescript fields to LHpi.DoImport
- adapted to new sitescripts and confirmed that nothing broke in the old ones
- DOING format logging more readable
- reorganized LHpi.sets cardcount and variants
- for kicks and giggles, have SAVETABLE generate a csv usable by woogerboys importprices :-)
- 2.2
- fixed handling of unset cardcounts
- updated for MA 1.5.2.264b (Dragon's Maze cardcount added)
- fixed savepath handling for new versioning scheme
- 2.3
- updated for MA 1.5.2.267 (M14 and MMA added)
- 2.4
- added Theros
- 2.5
- changed site.BCDpluginName(name, setid) to site.BCDpluginPre(card, setid) and site.BCDpluginCard(card, setid) to :site.BCDpluginPost(card, setid)
- fixes to Token handling
- GetSourceData now actually preserves all data from site.ParseHtmlData
- adjusted handling of multiple pages per set
- 2.6
- added BNG and Commander2013
- started adding missing special and promo sets
- expanded version tables for old expansions
- count foiltweak events
- minor improvements and fixes
- 2.7pre1
- fix handling of site.ParseHtmlData supplied sourcerow.* data
- pass supImportfoil from MainImportCycle to BuildCardData
- improved card dropping in BuildCardData
- ListSources now drops foilonly urls if foil import is deselected
- 2.7
- externalized static set data to LHpi.Data
- look for LHpi and LHpi.Data in Prices\lib\
- set default values for unset sitescript options
- fixed luadoc comments
- BuildCardData drops unwanted foil/nonfoil cards/prices to make sure user wishes are honoured
- 2.8
- DoImport: added savepath writable check to SAVECSV=true
- don't assume fruc[1] is foil and all other frucs are nonfoil
- site.fruc: add booleans to have each fruc identify itself as foil and/or nonfoil
- site.BuildUrl: infix changed from site.frucs[frucid] to site.frucs[frucid].url
- ProcessUserParams: disable unwanted frucs in site.sets at runtime, similar to how unwanted langs are already handled
- ListSources: removed check for url.foilonly, ProcessUserParams should have taken care of it.
- BuildCardData: removed @param #boolean urlfoil
- BuildCardData: make sure userParams are honoured, even when preset data is present, by setting unwanted prices nil before returning the card
- BuildCardData: BCDplugins should know everything BCD knows, so we're passing them importfoil, importlangs
- GetSourceData: now expects site.ParseHtmlData to return card(s) wrapped in a container table
- new STRICTCHECKEXPECTED option
- Toutf8: new optional parameter enc
- string.format'ed all nontrivial Log(string)s
- read static language fields from LHpi.Data.languages
- shortened site.langs
- hotfix: two bugs fixed (thanks to Bloodnut for reporting them)
- 2.9
- (all changes are transparent to sitescripts)
- fixed and improved MergeCardrows (was broken if importfoil=="N" or "O" - reported by Bloodnut)
- got rid of most global variables
- curhtmlnum
- local in MainImportCycle
- persetcount
- local in MainImportCycle
- BuildCardData also returns #boolean namereplaced, #boolean foiltweaked
- SetPrice also returns #table psetcount, #table failcount
- totalcount,setcountdiffers
- local in DoImport
- MainImportCycle returns #table totalcount, #table setcountdiffers
- curhtmlnum
- misc. small improvements to code and/or comments
- fixed loading of data file from deprecated "Prices" location
- fixed SAVETABLE folder writable check
- fixed logging resultregex finds
- LHpi.Log now only uses ma.Log when loglevel<0 is used
- LHpi.SetPrice adds abs(expected-retval) to failcount
- 2.10
- os.clock() loglevel 2 if os is available
- fix default logfile "LHpi.Log" -> "LHpi.log"
- more intuitive boolean option names
- DEBUGSKIPFOUND -> DEBUGFOUND
- STRICTCHECKEXPECTED -> STRICTEXPECTED
- legacy conversion for old sitescripts
- merge site.variants with Data.variants, same with foiltweak
- 2.11
- "(Version %d)" variant suffix from tcgplayer unified
- Log library version after loading
- 2.11hotfix
- BuildCardData now works with foil/nonfoil only when only one language is selected. (thanks to Bloodnut for reporting)
- 2.12
- BuildCardData
- replace Simplified Chinese Basic Lands' names for variant checking if names[9]
- improved card.drop logging
- remove trailing 0s from collector numbers
- MainImportCycle
- try to log number of unneeded pages for each set (to help optimize LHpi.mtgmintcard.lua)
- FillCardsetTable
- now keeps card.names table if VERBOSE, since I cannot properly debug non-latin cards otherwise (SZH in LHpi.mtgmintcard.lua)
- DoImport
- removed legacy conversion of STRICTCHECKEXPECTED to STRICTEXPECTED
- removed legacy conversion of DEBUGSKIPFOUND to DEBUGFOUND
- call site.SetExpected if CHECKEXPECTED
- 2.13
- DoImport
- finetune site.expected[setid].pset defaults according to site.expected.EXPECTTOKENS,site.expected.EXPECTNONTRAD and site.expected.EXPECTREPL
- increased default dataver from 2 to 5
- BuildCardData
- set card.object type
- FillCardsetTable
- also fill with objtype
- MergeCardrows
- merge objtypes
- SetPrice
- look for special variant names "token", "nontrad", "replica", "insert" and transform them into object types for ma.SetPrice
- 2.14
- cardcount.nontr renamed to cardcount.nontrad
- GetSourceData split into GetSourceData and ParseSourceData.
- this change is completely transparent to existing sitescripts, but having GetSourceData seperated allows to use it from the sitescript for special cases.
- LHpi.GetSourceData( sourceurl,urldetails )
- defer fetching to sitescript for oauth
- now returns #string sourcedata
- converts url to filename if OFFLINE
- fixed savepath==nil handling (needed when called before DoImport has run)
- Log https status if not ok
- LHpi.ParseSourceData( sourcedata,sourceurl,urldetails )
- does the parsing previously done by GetSourceData and returns the #table sourceTable
- shortcut return nil for sourcedata==nil
- now strictly expect site.ParseHtmlData to return card.price as #number (and not #table, as was possible before)
- do not sanitize nor check card.regprice or card.foilpriceBuildCardData
- counts namereplace and foiltweak even if card is dropped
- BuildCardData
- remove "(nontrad)","(replica)","(plane)","(scheme)","(conspiracy)" and
- "(oversized)" card.name infixes when no longer needed
- set objecttype replica for 105,106,69
- improved Token suffix handling
- keeps "Replica" suffix for variants
- removes "Insert" suffix
- keep (oversized) in set [40]
- SetPrice
- objtype infixe handling improved
- Logtable
- no longer strictly requires #table arguments
- MainImportCycle
- fixed CHECKEXPECTED handling for semi-available languages
- DoImport
- changed site.expected.EXPECTTOKENS to site.expected.tokens
- changed site.expected.EXPECTNONTRAD to site.expected.nontrad
- changed site.expected.EXPECTREPL to site.expected.replica
- all three can be #boolean (like before) or #table { [langid]=#boolean,... }
- DoImport passes supImportfoil,supImportlangs,supImportsets to site.SetExpected
- fixed expected defaults and checks
- removed some leftover commented-out code
- improved some comments and log msgs
- 2.15
- Split Initialize() from DoImport(importfoil , importlangs , importsets)
- always run Initialize() before returning LHpi library table.
- scriptname,savepath,logfile change from global to local (in site and LHpi)
- Log now filters out VERBOSE and DEBUG lines by loglevel param to save "if DEBUG then Log() end" conditionals
- those conditionals have been cut and loglevel has been set for all existing LHpi.Log calls
- DEBUGVARIANTS=true no longer sets DEBUG=false, instead remembers and restores previous DEBUG state
- new LHpi.OAuthEncode(s) moved here from mkm-helper
- workdir fixes
- 2.16
- removed leftover unused code
- removed deprecated parameter offline from site.BuildUrl calls
- site.resultregex findings (if any) logged in MainImportCycle instead of GetSourceData
- no longer check for Data in deprecated location
[edit] LHpi.Data
- 1
- initial release, split from LHpi library 2.6
- added all Special Sets
- 2
- (all changes transparent to sitescripts)
- minor improvements to some special sets' variant
- misc. small improvements to code and/or comments
- added 636:Salvat 2011
- 3
- added:
- 806 Journey into Nyx
- 805 Duel Decks: Jace vs. Vraska
- 804 Challenge Deck: Battle the Horde
- 803 Challenge Deck: Face the Hydra
- small fixes to some variant tables
- 4
- added: 235,807,808,105,106
- fixed: 320,490,590,139,787,26,31,30,10,740,390
- added lots of TODO markers
- 5
- renamed Data.sets[setid].cardcount.overs to Data.sets[setid].cardcount.repl
- add "Token" suffix to variant tables to enable objtype=2 in LHpi.BuildCardData
- add Nontrad,Replica variants to 807,769,787,70,761,786,784,782,801,778,23,21,20
- new sets 809,810,811,812,813
- fix cardcount, variant and/or foiltweak tables for 807,812,31,30,25,23,21,10
- 6
- new 52,53,55,814,815,816,817,818,819,820,821,822
- added lang [17]="Phyrexian"
- fixed names for 21,40,814
- variant table for 15,22,25,26,27,33,40,45
- foiltweak table for 22,26,33
- cardcount fixed/updated in 9,105,106,762,791,800,813,816,30,22,21,26
- fixed/updated variants for 43,69,201[ITA],758,764,766,781,787,800,813,22
- commented variants in 180,140,110,100,90,280,230,220,170,260,105,25
- foiltweak fixed/updated in 813
- 7
- new 825,823,824,826
- fixed variants for 822,819,25
- updated cardcount for 822,22
- 8
- variant table for 635 (Salvat Magic Encyclopedia; I'm glad that's finally done)
- add tla (three letter abbreviations) to all sets
- updated/fixed variants for 52,25,10
- updated/fixed cardcount for 791,53,52,50,21,10
- 9
- vupdated/fixed cardcount for 26
[edit] LHpi.dummyMA
- 0
- initial release as part of LHpi-v2.0.zip
- moved to seperate file instead of including it in LHpi lib zip
- 0.2
- Backwards compatible with Lua5.1
- tested with seperated LHpi.Data
- 0.3
- misc. small improvements to code and/or comments
- patch paths, because dev src has been moved from MA\Prices
- dummy.fakesitescript()
- 0.4
- execution timer in main()
- global variables changed to dummy.*
- new dummy.performancetest(repeats,script,impF,impL,impS,timefile)
- more comfort, less code duplication
- 0.5
- optional object type parameter in ma.SetPrice
- new sets 801,803,804,805,806,807,808,809,810,811,812,813
- 0.6
- new dummy.CompareDummySets(mapath,libver)
- new dummy.CompareDataSets(mapath,libver,dataver)
- new dummy.CompareSiteSets (moved from LHpi.magickartenmarkt.lua)
- new dummy.ListUnknownUrls(expansions,missing,file) (moved from LHpi.mkm-helper.lua)
- error handling in ma.GetFile and ma.Putfile
- for libver > 2.14 and Lua 5.2, use dofile instead of homemade loader
- CompareDataSets compares with dummy's set tables instead of Database/Sets.txt
- use global workdir as introduced in 2.15
- added 52,53,55,814,815,816,817,818,819,820,821
- fixed dummy.forceEnv
- fixed savepath handling
- deprecated dummy.loadlibonly and dummy.loadscript
- dummy.mergetables no longer changes param #table teins
- update helper functions ouput to log instead of stdout
- ma.GetUrl now uses luasockets instead of returning nil
- 0.7
- can run in helper mode, doing nothing but providing ma.* namespace
- renamed from dummyMA.lua to LHpi.dummyMA.lua
- added LHpi.magickartenmarkt.lua and LHpi.mkm-helper.lua to script selection
- ma.PutFile and ma.GetFile only print params if DEBUG
- added 825,826,824,823
- 0.8
- improved LE BOM handling
- function names CamelCased
[edit] external dependencies
- 1
- initial release for dummyMA-v0.6
- LuaSockets to implement ma.GetUrl
- 2
- release for LHpi.magickartenmarkt-v2.16.8.2
- add support for OAuth,SSL,JSON
[edit] LHpi.sitescriptTemplate
- 2.0
- initial release as part of LHpi-v2.0.zip
- 2.1
- use LHpi-v2.1
- let LHpi lib set savepath
- moved to seperate file instead of including it in LHpi lib zip
- 2.2.1
- use LHpi-v2.2
- 2.5.1
- use LHpi-v2.5
- 2.6.1
- use LHpi-v2.6
- pre-populated site.sets with all sets
- added LOGFOILTWEAK option
- 2.7.1.1
- use LHpi-v2.7 and LHpi.Data-v1
- load lib from \lib subdir
- cleaned up luadoc and improved documentation
- 2.8.1.1
- uses LHpi-v2.8 and LHpi-Data-v1
- site.frucs: new format
- sample BuildUrl adapted to new fruc format
- site.BCDplugins: added importfoil, importlangs parameters
- site.ParseHtmlData: now returns a container, to allow multiple cards
- new option STRICTCHECKEXPECTED
- 2.9.2.8
- too much simplification prevented possible functionality: reintroduced site.langs[langid].id field for reverse lookup
- split script version from scriptname to ease possible future change to versioning scheme
- misc. small improvements to example code and/or comments
- 2.10.2.9
- lib 2.10
- 2.12.4.10
- libver 2.12
- dataver 4
- BuildUrl: added optional fields to urldetails luadoc
- new (optional) field: site.pagenumberregex
- wrap site.expected in site.SetExpected()
- add missing sets
- 2.13.5.11
- new EXPECTNONTRAD and EXPECTREPL options to site.expected
- new sets 809,810,811,812,813
- new STRICTOBJTYPE option
- 2.14.5.12
- removed url to filename changes that are done by the library if OFFLINE
- 2.15.6.13
- script configuration variables local instead of global
- workdir support
- site.LoadLib split from ImportPrice
- site.Initialize loads LHpi lib if not yet available (needed if ImportPrice is not called)
- keep LHpi library table if already present
- no longer check for lib and Data in deprecated location
- removed site.CompareSiteSets
- optional site.BuildUrl("list") behaviour
- new optional function site.FetchExpansionList
- site.BuildUrl supports multiple urls per set
- 2.16.8.14
- merged mkm branch
- fixed some comments
- added 825,823,824,826
- defaults for site.expected[sid].namereplaced and site.expected[sid].foiltweaked
[edit] LHpi.magickartenmarkt
- 2.16.8.1
- unreleased
- 2.16.8.2
- initial release
- split mkm-helper
- 2.16.9.3
- updated expected for 30,26
- site.priceTypes format changed, descriptions added
- on HTTP errors, site.FetchSourceDataFromOAuth prints/logs errormsg and returns nil,status
- mkm price type choice in site.* namespace
- 2.16.9.4
- Properly initialise requestCounter file if none exist (thanks to Danatar for reporting and testing)
[edit] LHpi.mkm-helper
- 2.16.8.1
- initial release
- 2.16.9.2
- GetSourceData gracefully skips set on http errors
- table dataStaleAge { "default"=#number, #number setid=number, ... } instead of #number DATASTALEAGE
- new mode.forcerefresh
- new mode.checkstock (experimental, unpolished and largely undocumented)
- new helper.CheckStockPrices(maStock, mkmStock)
- new helper.GetStock()
- new helper.LoadCSV( filename )
- check dummy.version
- function names CamelCased
- 2.16.9.3
- don't update dataAge if set was aborted (i.e. 429)
- changed mode.checksales to mode.checkstock (now matching the documentation)
- Properly initialise timestamp file if none exist (thanks to Danatar for reporting and testing)
[edit] LHpi.tcgplayerPriceGuide
- 2.0
- initial release
- just adapted Golob's avsets to LHpi syntax and tested basic functionality
- 2.1
- updated for MA v1.5.2.264
- use LHpi-v2.1
- let LHpi lib set savepath
- tested all core and expansions up to gatecrash, also 3 portal sets and both "Un-sets"
- added namereplace table and expected card counts
- commented out remaining untested sets
- 2.2.1
- Updated for MA 1.5.2.264b (includes Dragon's Maze)
- use LHpi-v2.2
- 2.3.1
- use LHpi-v2.3
- added MMA and M14
- 2.4.1
- added Theros
- lots of Ae -> Æ replacements
- 2.5.1
- changed BuildUrl to match lib 2.5
- 2.6.1
- use LHpi-v2.6
- added BNG,FTV:20,DD:HvsM,DD:SvT,DD:VvK,DD:EvT,DD:DvD,Beatdown
- now with less expected fails and slightly better support for card variants
- 2.7.1.1
- use LHpi-v2.7 and LHpi.Data-v1
- load lib from \lib subdir
- cleaned site.sets' frucs
- added an option to just copy English card's prices to other languages
- 2.8.1.1
- use LHpi-v2.8
- shortened site.langs
- simplified site.frucs,site.sets to single fruc
- hotfix: set options to default values
- 2.9.2.10
- synchronized with template
- drop all "SOON" prices (higher maintenance for site.expected,but less "0"s in averaging)
- updated site.expected (apparently, they don't like Boros: a sizable portion is Plains and Mountains!?)
- 2.11.3.11
- JOU
- 2.12.4.12
- added 808, most special and promo sets
- TODO-marked all others
- updated expected and namereplace tables
- synchronized with template
- 2.13.5.13
- new sets 810,812,813
- removed oversized handling from BCDPluginPre
- fix 100,170,801,787,600,380,10,23
- add "Token" suffix to replacement names
- update site.expected
- synchronized with template
- 2.13.5.14
- added 814
- 2.14.5.14
- removed url to filename changes that are done by the library if OFFLINE
- 2.15.6.14
- added 816,818,815,817,819,820,821,40,32,27,25,23,22,21,822,340,55,45,822
- new features from template/mkm branch
- site.BuildUrl
- supports multiple urls per set
- set url.details.setid
- special handling for
- Dragonfury cards
- unusual lang cards
- updated/fixed/added variants,namereplacements,expected
- 2.15.7.15
- added 825,823,824,826
- fixed namereplace 822,808,110,100,90,751,570,220,170,820,819,814,807,801,796,787,600,45,23,22
- 2.16.8.16
- synced with template
- updated site.namereplace for 825,220,170,820,40
- updated site.expected
[edit] LHpi.magicuniverseDE
- 2.0
- initial release
- 2.1
- updated for MA v1.5.2.264
- use LHpi-v2.1
- let LHpi lib set savepath
- 2.2
- German Nemesis added
- 2.2.1
- Updated for MA 1.5.2.264b (includes Dragon's Maze)
- use LHpi-v2.2
- 2.3.1
- use LHpi-v2.3
- preemptively added M14
- 2.5.1
- preemptively added Theros
- removed lots of Ae -> Æ replacements
- changed BuildUrl to match lib 2.5
- 2.5.2
- fixed Theros
- 2.6.1
- use LHpi-v2.6 (BNG)
- 2.8.1.1
- uses LHpi-v2.8 and LHpi-Data-v1
- hotfix: set options to default values
- 2.9.2.10
- synchronized with template
- updated site.expected
- 2.11.3.11
- JOU
- 2.12.4.12
- added 801,808
- updated 180,791,806
- fixed site.langs
- synchronized with template
- 2.13.5.13
- new sets 807,813
- removed oversized handling from BCDPluginPre
- add "Token" suffix to replacement names
- update site.expected
- fix 140,420,801
- synchronized with template
- 2.13.5.14
- addded 813
- 2.14.5.14
- removed url to filename changes that are done by the library if OFFLINE
- 2.15.6.14
synchronized with template fixed/updated site.regex
[edit] LHpi.mtgmintcard
- 2.0
- initial release
- 2.1
- updated for MA v1.5.2.264
- use LHpi-v2.1
- let LHpi lib set savepath
- 2.2.1
- Updated for MA 1.5.2.264b (includes Dragon's Maze)
- use LHpi-v2.2
- 2.2.2
- updated expected dropcounts
- 2.3.1
- use LHpi-v2.3
- added MMA and M14
- 2.5.1
- added Theros
- changed BuildUrl (crudely) to multiple pages per set
- lots of small fixes
- 2.6.1
- use LHpi-v2.6
- added BNG,FTV:20,DD:HvsM,DD:SvT,DD:VvK,DD:EvT,DD:DvD,Beatdown,Unhinged,Unglued,Portal,Chronicles
- 2.8.1.1
- uses LHpi-v2.8 and LHpi-Data-v1
- hotfix: set options to default values
- 2.9.2.9
- synchronized with template
- updated site.expected
- 2.10.2.10
- lib 2.10
- foiltweak[setid].override=true
- 2.11.3.11
- JOU, JaceVsVraska
- 2.12.4.12
- added 807,808
- BuildUrl and tables changed for new site structure
- site.regex and ParseHtmlData changed for new raw format
- BCDPluginPre now handles "SZH (ENG)" card names
- added site.pagenumberregex
- updated all expected counts
- synchronized with template
- 2.13.5.13
- new sets 809,810,812,813
- removed oversized handling from BCDPluginPre
- add "Token" suffix to replacement names
- update site.expected
- fix 330,801,787,600
- synchronized with template
- 2.13.6.14
- added 814
- 2.14.5.14
- removed url to filename changes that are done by the library if OFFLINE
- 2.15.6.14
- synchronized with template
[edit] LHpi.trader-onlineDE
- 2.0
- unreleased
- 2.1
- updated for MA v1.5.2.264
- rewrote BuildURL and ParseHTMLData, as site was changed in the meantime.
- 2.2
- German Nemesis added
- 2.2.1
- Updated for MA 1.5.2.264b (includes Dragon's Maze)
- use LHpi-v2.2
- 2.3.1
- use LHpi-v2.3
- added MMA and preemptively M14
- 2.5.1
- preemptively added Theros
- lots of Ae -> Æ replacements
- use new 2.5 BuildCardDataPlugins for improved token handling
- changed BuildUrl to match lib 2.5
- 2.5.2
- fixed THS
- 2.6.1
- use LHpi-v2.6 (BNG)
- added BNG and all available special sets
- 2.8.1.1
- uses LHpi-v2.8 and LHpi-Data-v1
- hotfix: set options to default values
- 2.9.2.10
- synchronized with template
- updated site.expected
- 2.11.3.11
- JOU, JaceVsVraska
- 2.12.4.12
- added 801,807,808
- updated expected 100,680,690,800
- synchronized with template
- 2.13.5.13
- new sets 808,812,813
- add "Token" suffix to replacement names
- update site.expected
- fix 784,801,600
- synchronized with template
- 2.13.6.13
- added 814
- 2.14.5.13
- removed url to filename changes that are done by the library if OFFLINE
- 2.15.6.14
- synchronized with template
[edit] LHpi.mtgpprice.com
- 2.9.2.1
- initial release, unfinished
- 2.11.3.3
- first semi-official release
- still largely unmaintained
- JOU,DDM
- 2.12.4.3
- site.regex,ParseHtmlData fixed for new site layout
- added JOU,DDM,CNS,M15, special sets
- synchronized with template
- some more of namereplacement for basic land variants
- fixed offline urls
- 2.13.5.4
- new set 813
- fix 250,180 (except basic lands)
- synchronized with template
- 2.13.6.5
- added 814
- 2.14.5.5
- removed url to filename changes that are done by the library if OFFLINE
- 2.15.6.5
- synchronized with template
[edit] License
[edit] LHpi
The LHpi library and all its sitescripts are released as open source under the GNU General Public License. Please note that LHpi is thusly licensed seperately from Magic Album proper, which to date is closed source.
[edit] third party libraries
LHpi.dummyMA, LHpi.magickartenmarkt and LHpi.mkm-helper also require external libraries. Those are taken from the following projects:
[edit] Lua
lua52.exe and lua52.dll are distributed under the MIT license.
[edit] LuaSocket
LuaSocket is © 2004-2007 Diego Nehab and distributed under the same terms and conditions as the Lua language.
[edit] dkjson
dkjson is © 2010-2013 David Heiko Kolf and distributed under the same conditions as the Lua interpreter.
[edit] LuaOAuth
LuaOAuth is © 2010 Ignacio Burgueño and distributed under the MIT License. It contains code from lua-multipart-post © 2012-2013 by Moodstocks SAS under the same license.
[edit] base64.lua
base64.lua is © 2009 by Alex Kloss, licensed under the terms of the LGPL2.
[edit] LuaCrypto
LuaCrypto is © 2006 Keith Howe and uses the same license as Lua 5.0.
[edit] LuaSec
LuaSec 0.5 is © 2006-2014 Bruno Silvestre and distributed under the MIT License
[edit] OpenSSL
OpenSSL © 1998-2011 The OpenSSL Project, licensed under OpenSSL License. This product includes cryptographic software written by Eric Young (eay@cryptsoft.com) and software written by Tim Hudson (tjh@cryptsoft.com), licensed under the Original SSLeay License.
[edit] Cryptograpy export notice
Import and/or use of strong cryptography software is illegal in some parts of the world. If this applies to you do not download the external dependecies file. By downloading this software, you agree to and take full responsibility to abide by the relevant laws.
[edit] Other scripts
Just for reference: These unrelated pricing scripts exist
- "MTG Mint Card.lua" by Goblin Hero and Stromglad1 is included with Magic Album as a working script example.
- "Import Prices" by woogerboy21.