Log in
  Page  Discussion  View source  History

LHpi

LHpi is LordHelmchen's price import library and sitescripts for Magic Album.

LHpi is a collection of lua scripts to be used by Magic Album, but is not part of Magic Album and thus updated and supported seperately.

Discuss, ask for help and leave feedback at the corresponding Forum Thread.

Contents

Sitescripts

The sitescripts are the actual price import scripts, written in lua. They use the Magic Album Lua API and the LHpi library to query and parse a website and import the price information into Magic Album. Most of the program logic is contained in the library, so the sitescript needs to hold only site-specific information and a few functions that are not easily generalizable.

In theory, you should not be dependent on a new version once a new set is available: just add new entries to site.sets table in the sitescript. Being easy to update yourself was one of the design goals.

Sitescripts will need to provide

  • a table of supported sets with language and url infix fields
  • a table of supported languages with url infix field
  • a function to build urls
  • a function to parse the html raw data into data fields via regular expressions (and the regex itself)
  • optionally, they can also provide
    • functions for site specific data manipulation, in case the site has some strange formatting the library does not understand
    • a one-to-one name map for names that differ between the site and Magic Album's database
    • tables to map card names to picture versions, in case the one-to-one map and LHpi's defaults are not sufficient
    • a table with expected card count numbers for LHpi's sanity checks.

magicuniverse.de

LHpi.magicuniverseDE-v2.6.1.lua.zip
Sitescript version 2.6.1
used LHpi version 2.6
updated last on 2014-02-11
size 11.55 KiB

Magic Universe is a German site with German and English cards, priced in €.

This site started it all, since I wanted prices in local currency, and my FLG apparently uses them as price reference.

mtgmintcard.com

LHpi.mtgmintcard-v2.6.1.lua.zip
Sitescript version 2.6.1
uses LHpi version 2.6
updated last on 2014-02-11
size 9.74 KiB

Functionally the same as "MTG Mint Card.lua" by Goblin Hero and Stromglad1.

When I wanted to generalize my script for easy addaption to other sites, this provided me with a good baseline, since I could search MA for cards that my script missed but should have set. Also, the site is easy to querry and parse. They do have a currency chooser, so I started by trying to make a mtgmintcard-Euro script. I failed, but salvaged the wreck for default $ prices.

magic.tcgplayer.com PriceGuide

LHpi.tcgplayerPriceGuide-v2.6.1.lua.zip
Sitescript version 2.6.1
uses LHpi version 2.6
updated last on 2014-02-11
size 12.74 KiB

Tcgplayer Price Guide claims to accumulate prices from 75 online vendors. They are priced in $. Their tables give a high, low and average price; the sitescript contains a variable to select which one of those it should use.
As requested by Golob here. I have only adjusted the syntax of his avsets to LHpi and added the card count expectations table. Also, Promo and most special sets have not been tested yet and are commented out.
Note that cards with multiple versions are averaged in some sets and set to "variant 1"'s price in others, as the source list does not clearly identify the version. Even when it does, it is unclear if the entry without version suffix is meant to be all unspecified or version 1 only. As I wanted to keep this sitescript as basic as possible, I decided against guessing a real variant table and just added expected failcounts.
To change the column to import from, open the script in a text editor, find himelo = 3 which should be around line 126 and change it to himelo = 1 for high or himelo = 2 for medium prices. You can have #multiple copies ot the same script

trader-online.de

LHpi.trader-onlineDE-v2.6.1.lua.zip
Sitescript version 2.6.1
uses LHpi version 2.6
updated last on 2014-02-11
size 11.98 KiB

Trader Online is a German site with German and English cards, priced in €.

I found this site while looking for Token images, found it to be parseable by sitescript, and decided to use it as a test on how well the library was adapted to support additional sites.

The LHpi Library

LHpi-v2.6.lua.zip
Library version 2.6
Tested with Ma version 1.6.1.321
updated last on 2014-02-11
size 30.33 KiB

It is not expected to be fast, but as reliable as possible:
It first compares the user requested sets and languages with what the sitescript has available and builds a table of urls to querry, seperated by set.
Then it loops though the sets that are both available and selected:
The raw html data is fetched, parsed and entered into an intermediate table of all prices for the current edition, merging regular and foil prices to minimize calls to ma.SetPrice.
Should the same card (defined by set,card,language,(picture-)version ) be encountered again, it compares the prices. If they are equal, the new price is discarded; if they differ (Yes, magicuniverse.de's database is quirky), it uses the average price. Cards that are suffxed with a condition description are currently disregarded.
Once that cardsetTable is completed (all urls belonging to the set are parsed), it sends the prices to MA. Finally, it compares the count of set prices with the number of cards in the set (or, if given, the expectation given in the sitescript).

Logging to a seperate file is now enabbled per default (but still configurable by the variable SAVELOG in the script), since ma.PutFile(file,data,0) behaviour bug was fixed. The log is overwritten on subsequent re-runs of the script.

Installation and usage

  1. Download at least one sitescript and the LHpi libray version the sitescript uses.
  2. Unzip into the "Prices" subdirectory of your Magic Album installation.
  3. (optional) If you want to use the SAVEHTML feature, you need to create an empty subdirectory, named like the sitescript without version suffix (for example "Prices\LHpi.magicuniverseDE").
  4. (re)start Magic Album and use the sitescript (not the library) from Magic Album's Price Manager dialog.

If something does not happen as expected

  • make sure the script is saved utf-8 encoded. It's customary for lua scripts to be saved ANSI encoded, but as MA expects utf input, this is what I considered the most wysiwyg approach to set cards with special characters in their names.
  • open the lua file in a text editor and change "VERBOSE = false" to "VERBOSE = true" (line 33). This will make the script provide feedback to help track down any problems (or just to see what it was doing) and also enable additional checks. The logging will be sent to "[MA Folder]\Prices\[scriptname].log".

I'll try to answer any questions you might have. If the script misbehaves please follow the two steps above and attach a copy of the .log to your post.

Development

sitescriptTemplate

LHpi.sitescriptTemplate-v2.6.1.lua.zip
Sitescript version 2.6.1
uses LHpi version 2.6
updated last on 2014-02-11
size 7.63 KiB

This is just an empty, commented template to help write new sitescripts.

dummyMA.lua

dummyMA.zip
version 0.1
updated last on 2013-04-10
size 5.28 KiB

Magic Album API dummy to test and debug LHpi within an IDE (such as eclipse) and without needing Magic Album.

Planned features

LHpi-v2.7pre1.lua.zip
version 2.7-pre1
updated last on 2014-02-11
size 30.55 KiB

Current work-in-progress version.

in no particular order

  • Promo and special sets for magicuniverseDE,mtgmintcards, tcgplayerPriceguide
    • add all special and promo sets (cardcount,variants,foiltweak) to LHpi.sets
  • tweak log : better differentiation of log levels (normal,verbose,debug); rewrite normal-level logentries to be more pleasing to read.
    I'd love to get users' feedback on this: how silent should normal level be; what should not be logged unless in debug mode?
  • add a few progressbar updates?
  • no longer silently assume that fruc is {foil,rare,uncommon,common}
    • that means ProcessUserParams should not use importfoil,importlangs to nil parts of site.sets[setid].fruc and site.sets[setid].langs
    • instead, apply [url].foil=y|n|o and [url].lang={[langid]=boolean,...} in site.BuildUrl, then use importfoil,importlangs to
      • have ListSources drop unwanted urls to minmize network load. partially done

*** have BuildCardData drop unwanted cards/prices to make sure user wishes are honoured done **** make BuildCardData aware of user's choices done

    • if taken further, passing even importsets to BuildCardData could sort out the mess that is magicuniverse's promo page.
  • to anticipate sites that return "*CARDNAME*REGPRICE*FOILPRICE* instead of "*CARDNAME*PRICE*FOILSTATUS*": have site.ParseHtml return a collection of cards, similar to site.BuildUrl

* fix handling of site.ParseHtmlData supplied sourcerow.* data done

Possible features

in no particular order

  • seperate library (code) and LHpi.sets (data) into 2 files ?
    Would make updates/versioning cleaner by distinguishing between new set info,variant tables (data) and further generalization,bugfixing (code)
    • ideally, most information in LHpi.sets could be read from MA via API getter functions.
  • add hardcoded exchange rate for crude currency conversion between parsing and importing.

* handle conflicting prices differently. Is averaging ok, or should the script discard one (which?) of the prices and/or throw an error ? Until further notice, I'm ok with how averaging works.

  • import prices from condition-suffixed cards instead of dropping them.
    • patch to accept entries with a condition description if no other entry with better condition is in the table:
      • buildCardData will need to add condition to carddata
      • global conditions{} to define priorities
      • then fillCardsetTable needs a new check before overwriting existing data
  • add another sitescript (and adapt library if needed)
  • only magicuniverse uses multiple frucs. maybe it's time to revert back to foil=Y|N|O and move common|uncommon|rare url building into the sitescript

Problems

  • can't set the sitescript's name dynammically. 
--[[FIXME the dynamic approach myname does not work, ma.GetFile returns nil for its own log :(
do
	--local _s,_e,myname = string.find( ma.GetFile("Magic Album.log"), "Starting Lua script .-([^\\]+%.lua)$" )
	if myname then
		scriptname = myname
	else -- use hardcoded scriptname as fallback
		scriptname = "LHpi.magicuniverseDE-v2.0.lua" -- should always be equal to the scripts filename !
	end
end
--]]
  • number of cards in set must be hardcoded into library. An [url=http://www.slightlymagic.net/forum/tracker.php?p=2&t=303 ma.GetCardsInSet( setid,langid,regular|tokens )] would help, but the numbers are easy to hardcode anyways.
  • can't use mtgmintcard to get € prices. see [1] for details.
  • cards with quotation marks in their name currently confuse the script. Affected cards are
    • “Ach! Hans, Run!” (UNH)
    • Kongming, “Sleeping Dragon” (PTK)
    • Pang Tong, “Young Phoenix” (PTK)
  • Can't distinguish between regular and oversized versions of the same card. probably affects all Commander sets.

Hints for developers

Some information that might be useful if you want to write/improve sitescripts. Please add questions related to developing here, so I know what needs to be documented.

multiple copies ot the same script

You can have multiple versions of the same script, but you need to make sure scriptname = "LHpi.SCRIPTNAME-v" .. libver .. ".x.lua" (around line 70) is changed to reflect the script's filename. Otherwise, they will overwrite each other's logfile and html raw data. Of course, this could be intentional, so keep the name identical if that is what you want.

procedure of function calls

MA calls sitescript.ImportPrice
sitescript.Importprice loads the library, then calls LHpi.DoImport
LHpi.DoImport calls LHpi.ProcessUserParams and initializes
LHpi.DoImport calls LHpi.ListSources
 LHpi.ListSources loops through sets,langs,frucs and calls site.BuildUrl
LHpi.DoImport calls LHpi.MainImportCycle
LHpi.MainImportCycle loops through sets,urls and calls LHpi.GetSourceData
 LHpi.GetSourceData loops through matches with site.regex and calls site.ParseHtmlData, then returns sourceTable
LHpi.MainImportCycle loops through sourceTable and calls LHpi.BuildCardData
 LHpi.BuildCardData calls site.BCDpluginPre and site.BCDpluginPost and returns newcard
 LHpi.MainImportCycle calls LHpi.FillCardsetTable with newcard
  LHpi.FillCardsetTable might call LHpi.MergeCardrows
  then adds a (merged) row to cardsetTable
(optionally) LHpi.MainImportCycle calls LHpi.SaveCSV once cardsetTable is complete
LHpi.MainImportCycle loops through cardsetTable and calls LHpi.SetPrice

dropping cards

  • if LHpi.BuildCardData sets card.drop=true, LHpi.FillCardsetTable is skipped.
  • site.ParseHtmlData can set card.drop, which will be preserved by LHpi.BuildCardData.
  • if not card.name then card.drop = true
  • if string.find( card.name , "%(DROP[ %a]*%)" ) then card.drop = true
    • An example is in LHpi.magicuniverseDE.lua's site.BCDpluginPre. The advantage of changing the name above setting card.drop directly is that you can also add a reason for dropping to the name, which will be loggged if LOGDROPS=true.

version history

I decided to start all versions at 2.0, since it's the continuation of magicuniverseDEv1.6.lua, but with enough changes to warrant the next major version number. Do not expect revision numbers to stay synched. Starting with 2.2.1, sitescript minor version will conform to used library version, and the sitecripts will have their version suffixed. This should make it more clear which versions to keep and what can be deleted after updating. The sitescripts will know which library version they need and attempt to load the right one or exit gracefully.

pre-2.0

1.2 (Downloaded 22 times)
first public release
1.3 (Downloaded 20 times)
started externalizing site-specific data to seperate .config . exeptions for some sets still remain in the code.
1.4 (Downloaded 29 times)
sanity checks should be sane again :)
config externalization progressed far enough that adding new sets should be easy
1.5 (Downloaded 5 times)
added RtR to avsets
updated expected fail and drop counts
continued config externalization and script generalization efforts
moved gpl.txt from within the zip to seperate attachment
1.6 (Downloaded 1 time)
last release under the name magicuniverseDEv1.x.lua
added Gatecrash to avsets and expectedcounts (shop does not yet provide foil)
renamed .config to .lconf to ease syntax highlighting
got rid of utf-8 BOM
changed some comments to luadoc-like

LHpi lib

2.0
completely refactored, I'll call the externalization complete:
config became sitescript, old script (and work-in-progress mtgmintcardEURO script) became the library.
improved language handling
slightly improved conflict handling: same price is merged, different price is averaged.
2.1
added number of cards per set
if no expectation is defined in sitescript, expect all cards and 0 tokens to be set successfully
minor fix to debug loging
moved handling of undefined sitescript fields to LHpi.DoImport
adapted to new sitescripts and confirmed that nothing broke in the old ones
DOING format logging more readable
reorganized LHpi.sets cardcount and variants
for kicks and giggles, have SAVETABLE generate a csv usable by woogerboys importprices :-)
2.2
fixed handling of unset cardcounts
updated for MA 1.5.2.264b (Dragon's Maze cardcount added)
fixed savepath handling for new versioning scheme
2.3
updated for MA 1.5.2.267 (M14 and MMA added)
2.4
added Theros
2.5
changed site.BCDpluginName(name, setid) to site.BCDpluginPre(card, setid) and site.BCDpluginCard(card, setid) to :site.BCDpluginPost(card, setid)
fixes to Token handling
GetSourceData now actually preserves all data from site.ParseHtmlData
adjusted handling of multiple pages per set
2.6
added BNG and Commander2013
started adding missing special and promo sets
expanded version tables for old expansions
count foiltweak events
minor improvements and fixes
2.7pre1
fix handling of site.ParseHtmlData supplied sourcerow.* data
pass supImportfoil from MainImportCycle to BuildCardData
improved card dropping in BuildCardData
if importfoil == "n", drop foilonly urls in ListSources

magicuniverseDE

2.0
initial release
2.1
updated for MA v1.5.2.264
use LHpi-v2.1
let LHpi lib set savepath
2.2
German Nemesis added
2.2.1
Updated for MA 1.5.2.264b (includes Dragon's Maze)
use LHpi-v2.2
2.3.1
use LHpi-v2.3
preemptively added M14
2.5.1
preemptively added Theros
removed lots of Ae -> Æ replacements
changed BuildUrl to match lib 2.5
2.5.2
fixed Theros
2.6.1
use LHpi-v2.6 (BNG)

mtgmintcard

2.0
initial release
2.1
updated for MA v1.5.2.264
use LHpi-v2.1
let LHpi lib set savepath
2.2.1
Updated for MA 1.5.2.264b (includes Dragon's Maze)
use LHpi-v2.2
2.2.2
updated expected dropcounts
2.3.1
use LHpi-v2.3
added MMA and M14
2.5.1
added Theros
changed BuildUrl (crudely) to multiple pages per set
lots of small fixes
2.6.1
use LHpi-v2.6
added BNG,FTV:20,DD:HvsM,DD:SvT,DD:VvK,DD:EvT,DD:DvD,Beatdown,Unhinged,Unglued,Portal,Chronicles

tcgplayerPriceGuide

2.0
initial release
just adapted Golob's avsets to LHpi syntax and tested basic functionality
2.1
updated for MA v1.5.2.264
use LHpi-v2.1
let LHpi lib set savepath
tested all core and expansions up to gatecrash, also 3 portal sets and both "Un-sets"
added namereplace table and expected card counts
commented out remaining untested sets
2.2.1
Updated for MA 1.5.2.264b (includes Dragon's Maze)
use LHpi-v2.2
2.3.1
use LHpi-v2.3
added MMA and M14
2.4.1
added Theros
lots of Ae -> Æ replacements
2.5.1
changed BuildUrl to match lib 2.5
2.6.1
use LHpi-v2.6
added BNG,FTV:20,DD:HvsM,DD:SvT,DD:VvK,DD:EvT,DD:DvD,Beatdown
now with less expected fails and slightly better support for card variants

trader-onlineDE

2.0
unreleased
2.1
updated for MA v1.5.2.264
rewrote BuildURL and ParseHTMLData, as site was changed in the meantime.
2.2
German Nemesis added
2.2.1
Updated for MA 1.5.2.264b (includes Dragon's Maze)
use LHpi-v2.2
2.3.1
use LHpi-v2.3
added MMA and preemptively M14
2.5.1
preemptively added Theros
lots of Ae -> Æ replacements
use new 2.5 BuildCardDataPlugins for improved token handling
changed BuildUrl to match lib 2.5
2.5.2
fixed THS
2.6.1
use LHpi-v2.6 (BNG)
added BNG and all available special sets

sitescriptTemplate

2.0
initial release as part of LHpi-v2.0.zip
2.1
use LHpi-v2.1
let LHpi lib set savepath
moved to seperate file instead of including it in LHpi lib zip
2.2.1
use LHpi-v2.2
2.5.1
use LHpi-v2.5
2.6.1
use LHpi-v2.6
pre-populated site.sets with all sets
added LOGFOILTWEAK option

dummyMA

0
initial release as part of LHpi-v2.0.zip

0.1

moved to seperate file instead of including it in LHpi lib zip

License

The LHpi library and all its sitescripts are released as open source under the [GNU General Public License]. Please note that LHpi is thusly licensed seperately from Magic Album proper, which to date is closed source.

Other scripts

Just for reference: These unrelated pricing scripts exist

  • "MTG Mint Card.lua" by Goblin Hero and Stromglad1 is included with Magic Album as a working script example.
  • "Import Prices" by woogerboy21.
Retrieved from "https://www.slightlymagic.net/mw/index.php?title=LHpi&oldid=5186"