LordHelmchen's price import - BFZ and mkm

[LordHelmchen]

UPDATED 2013-04-10 (April 10th)

Until further notice I will cease rewriting this post over and over again. The information and files below are probably out-of-date. Please go to the LHpi's Wiki page for all documentation and download links.

ADDITIONAL NOTE 2014-02-11 (Feb 2nd)
The above note doesn't mean you should not post questions or praise in the thread, so please go ahead
Special thanks to randerscheinung, as him clicking "thanks" on each new release post was practically all feedback I received on my scripts for the last year

ADDITIONAL NOTE 2104-05-05 (May 5th)
Because there have been almost as much downloads of old versions in the last three months than there have been of the current versions, let me state this again:The information and files in in all but the most recent posts of this thread are probably out-of-date. Please go to the LHpi's Wiki page for all documentation and download links.The version numbers in the changelog are also download links that will get you the corresponding (old!) file.
Do post questions, bugs, improvement ideas, praise or other feedback in this thread.Or open/use the wiki's discussion page, if that seems more appropriate.

----
Post updated Feb.7th. The old text has been archived to the second post.

Greetings.

This is the release and discussion thread for LHpi (LordHelmchen's price import), my pricing script library and its sitescripts.
Supports all core and expansion sets up to Gatecrash out-of-the-box. In theory, you should not be dependent on a new version once a new set is available: just add new entries to site.sets table in the sitescript.

I decided to start all versions at 2.0, since it's the continuation of magicuniverseDEv1.6.lua, but with enough changes to varant the next major version number. Do not expect revision numbers to stay synched. The sitescripts will know which library version they need and attempt to load the right one or exit gracefully.

To use it you need to have both the library LHpi-v2.0.lua (or a later version, as required by the sitescript) and a sitescript.
Currently offered sitescripts are:

LHpi.magicuniverseDE-v2.0.lua.zip

(12.6 KiB) Downloaded 815 times

LHpi.magicuniverseDE-v2.0.lua | Open

LHpi.magicuniverseDE-v2.0.lua
German site with German and English cards, priced in €.
This site started it all, since I wanted prices in local currency, and my FLG apparently uses them as price reference.

LHpi.mtgmintcard-v2.0.lua.zip

(7.96 KiB) Downloaded 793 times

LHpi.mtgmintcard-v2.0.lua | Open

LHpi.mtgmintcard-v2.0.lua
Functionally the same as "MTG Mint Card.lua" by Goblin Hero and Stromglad1.
When I wanted to generalize my script for easy addaption to other sites, this provided me with a good baseline, since I could search MA for cards that my script missed but should have set.
Also, the site is easy to querry and parse. They do have a currency chooser, so I started by trying to make a mtgmintcard-Euro script. I failed, but salvaged the wreck for default $ prices.

LHpi.tcgplayerPriceGuide-v2.0.lua | Open

LHpi.tcgplayerPriceGuide-v2.0.lua
As requested by Golob here.
I have not checked his avsets, just adjusted the syntax to LHpi.
Does not contain card count expectations table for sanity checking. Also, Promo and special sets have not been tested yet.

LHpi.sitescriptTemplate-v2.0.lua | Open

LHpi.sitescriptTemplate-v2.0.lua
Just an empty, commented template to write new sitescripts. Included in library zip.

Update probability is higher for LHpi.magicuniverseDE-v2.0.lua than for the others, both because that is the one I use and because the other scripts (so far) are much simpler to update yourself.

It is not expected to be fast, but as reliable as possible:
It first compares the user requested sets and languages with what the sitescript has available and builds a table of urls to querry, seperated by set.
Then it loops though the sets that are both available and selected:
The raw html data is fetched, parsed and entered into an intermediate table of all prices for the current edition, merging regular and foil prices to minimize calls to ma.SetPrice.
Should the same card (defined by set,card,language,(picture-)version ) be encountered again, it compared the prices. When they are equal, the new price is discarded. If they differ (Yes, magcuniverse.de's database is quirky), it uses the average price.
Cards that are suffxed with a condition description are currently disregarded.
Once that cardsetTable is completed (all urls belonging to the set are parsed), it sends the prices to MA.
All the while doing more sanity checks than could be considered sane

Logging to a seperate file is now enabbled per default (but still configurable by the variable SAVELOG in the script), since ma.PutFile(file,data,0) behaviour bug was fixed by our Hero. The log is overwritten on subsequent re-runs of the script.

If something does not happen as expected
- make sure the script is saved utf-8 encoded. It's customary for lua scripts to be saved ANSI encoded, but as MA expects utf input, this is what I considered the most wysiwyg approach to set cards with special characters in their names.
- open the lua file in a text editor and change "VERBOSE = false" to "VERBOSE = true". This will make the script provide feedback to help track down any problems (or just to see what it was doing) and also enable additional checks. The logging will be sent to "[MA Folder]\Prices\[scriptname].log".
I'll try to answer any questions you might have. If the script misbehaves please follow the two steps above and attach attach a copy of the .log.

Advice on and ideas for improving the script are most welcome. See also the second post for plans and problems.

changelog | Open

1.2
1.2
first public release
(Downloaded 22 times)
1.3
started externalizing site-specific data to seperate .config . exeptions for some sets still remain in the code.
(Downloaded 20 times)
1.4
sanity checks should be sane again
config externalization progressed far enough that adding new sets should be easy
(Downloaded 29 times)
1.5
added RtR to avsets
updated expected fail and drop counts
continued config externalization and script generalization efforts
moved gpl.txt from within the zip to seperate attachment
(Downloaded 5 times)
1.6
last release under the name magicuniverseDEv1.x.lua
added Gatecrash to avsets and expectedcounts (shop does not yet provide foil)
renamed .config to .lconf to ease syntax highlighting
got rid of utf-8 BOM
changed some comments to luadoc-like
(Downloaded 1 time)
---------
LHpi-v2.0 and sitescripts
completely refactored, I'll call the externalization complete.:
config became sitescript, old script (and work-in-progress mtgmintcardEURO script) became the library.
Improved Language handling
slightly improved conflict handling: same price is merged, different price is averaged.

To install, unzip both LHpi-v2.x and at least one sitescript into the "Prices" subdirectory of your MA installation and restart MA.
All files belonging to old versions can be deleted. If you want to use the SAVEHTML feature, you need to create an empty subdirectory, named like the sitescript without version suffix (for example "Prices\LHpi.magicuniverseDE").

[LordHelmchen]

original first post archived here | Open

Greetings.
My price import script has reached a point where it might be considered useable, so it's time to share and enjoy.

It's not expected to be fast, but as reliable as possible: after the webpage has been parsed, the script builds an intermediate table of all prices for the currently imported edition, only sending the prices to MA after the table is complete, doing more sanity checks than could be considered sane

install instructions for 1.2: If you just want to see the prices in MA, drop the attached file into [MA folder]\Prices, rename it from "magicuniverseDEv1.2.lua.txt" to just "magicuniverseDE.lua" and run it from within MA.

The new Version finally reached the same amount of reliability as 1.2, so here's a version that should 'just work' again.
If anyone does need instructions on installing it, please do ask.
In theory, you should not be dependent on a new version once a new set is available: just add new entries to avsets and variants tables in the .config file.
Logging to a seperate file is disabled per default (but configurable by the variable SAVELOG in the script), as the log could potentially grow very large and ma.PutFile(file,data,0) behaviour does not match wiki.
If something does not happen as expected
- make sure the script is saved utf-8 encoded. It's customary for lua scripts to be saved ANSI encoded, but as MA expects utf input, this is what I considered the most wysiwyg approach to set cards with special characters in their names.
- open the lua file in a text editor and change "VERBOSE = false" to "VERBOSE = true". This will make the script provide feedback to help track down any problems (or just to see what it was doing) and also enable additional checks. The logging will be sent to "[MA folder]\Magic Album.log", but on the next MA update it will go into a seperate logfile, [MA Folder]\Prices\[scriptname].log".
I'll try to answer any questions you might have. If the script misbehaves please follow the two steps above, restart MA, run the script _once_ and attach a copy of the .log.

obsolete paragraph: If you want to be bleeding edge or learn from my mistakes, I'll asume you know how to handle [strikethrough].7z[/strikethrough] .zip and its contents

To install, unzip the contents of the .zip into the "Prices" subdirectory of your MA installation.

Advice on and ideas for improving the script are most welcome.

pre-2.0 first post archived here | Open

Post updated Jan.29th. The old text has been archived to the second post.

Greetings.
This is the release and discussion thread for my pricing script, currently using magicuniverse.de as data source.
Now supports Return to Ravnica and Gatecrash out-of-the-box (where box is zip).

It is not expected to be fast, but as reliable as possible: after the webpage has been parsed, the script builds an intermediate table of all prices for the currently imported edition, only sending the prices to MA after the table is complete, doing more sanity checks than could be considered sane
In theory, you should not be dependent on a new version once a new set is available: just add new entries to avsets and variants tables in the .config file.
A long term goal is to have universal script, and be able to configure it for other sites via the seperate .config file.

Logging to a seperate file is now enabbled per default (but still configurable by the variable SAVELOG in the script), since ma.PutFile(file,data,0) behaviour bug was fixed by our Hero.

If something does not happen as expected
- make sure the script is saved utf-8 encoded. It's customary for lua scripts to be saved ANSI encoded, but as MA expects utf input, this is what I considered the most wysiwyg approach to set cards with special characters in their names.
- open the lua file in a text editor and change "VERBOSE = false" to "VERBOSE = true". This will make the script provide feedback to help track down any problems (or just to see what it was doing) and also enable additional checks. The logging will be sent to "[MA Folder]\Prices\[scriptname].log".
I'll try to answer any questions you might have. If the script misbehaves please follow the two steps above, restart MA, run the script _once_ and attach a copy of the .log.


Advice on and ideas for improving the script are most welcome.

pre-2.0 second post archived here | Open

still working on it...
1.3 was kind of buggy compared to 1.2... the next version will be better and uploaded soon-ish
At least I think I corrected all bugs in the scripts own sanity checks, which made them return unappropriate thumbs-up half of the time

I'm in the process of externalizing everything that is specific to the site to the .config. The goal is to have universal script, and be able to configure it for other sites with a seperate .config file. I think it's progressing well, and once I have the infrastructure for sites with seperate pages for each language in place, will give it a shot at mtgmintcards, which does have a price-in-euro setting. (If I had seen that earlier, I'd just have changed the url in the original mtgmintcards script and be done with it... )

There's still one variable that I could not move to the .config, but maybe somebody knows why


Code: Select all
--[[ does not work, GetFile returns nil for its own log
local _s,_e,myname = string.find( ma.GetFile("\\Magic Album.log"), "Starting Lua script .-([^\\]+%.lua)$" )
]]

Duplicate checking and condition-suffix evaluation is still on the horizon, for now all suffixed entries are dropped and identical names will overwrite prior entries (or exit with error if in DEBUG mode)

Planned features:

• Promo and special sets for magicuniverse (as far as possible), mtgmintcards.
• tweak log : better differentiation of log levels (normal,verbose,debug); rewrite normal-level logentries to be more pleasing to read.
  I'd love to get users' feedback on this: how silent should normal level be; what should not be logged unless in debug mode?
• have unset expected count default to number of cards in set.
  not enough to release a 2.1 yet, but if you want it, change LHpi-v2.0.lua line number 1998 to
  

  Code: Select all

  if CHECKEXPECTED then
   if not site.expected then site.expected = {} end
   for sid,cardcount in pairs(LHpi.sets.cardcount) do
      if not site.expected[sid] then
      -- default for English cards only
         site.expected[sid] = { pset={ [1] = cardcount.reg } , failed={ [1]=0 } }
      end -- if
   end -- for
end -- if


Possible features:

• add hardcoded exchange rate for crude currency conversion between parsing and importing.
• handle conflicting prices differently. Is averaging ok, or should the script discard one (which?) of the prices and/or throw an error ?
• import prices from condition-suffixed cards instead of dropping them.
• add another sitescript (and adapt library if needed)

Problems:

• can't set the sitescript's name dynammically. see the archived second post for details
• number of cards in set must be hardcoded. An ma.GetCardsInSet( setid,langid,regula|tokens ) would help, but the numbers are easy to hardcode anyways.
• can't use mtgmintcard to get € prices. see here for details.

[LordHelmchen]

Greetings again.
Replying to my own posts starts to feel strange... but I think the new version is worth a bump, and the downloads for 1.4 compared to 1.6 and 1.5 suggest that either
a) editing the main post does not catch the eye as well as a new post
or b) even 1.4 worked so well and was easy enough to update that nobody needed the new version

Both above post were rewritten to reflect the new state, so please take a minute to glance over them again.

[LordHelmchen]

To prevent further confusion, I will cease rewriting the first two posts of the thread over and over again. Instead I created LHpi's Wiki page for documentation and download links. The thread shall still be used for dscussion and support, of course. I have also taken the liberty to add a sentence about LHpi to Magic Album's wiki page and a link to that from the wiki's Main Page.

There are a few changes to the library and the sitescripts, as well as a new sitescript. Files are attached here without further documentation, please see the wiki for details if you're unsure about what you will want to download.

[LordHelmchen]

Just need a second post due to 5 attachments per post limit.

[LordHelmchen]

Minor update for German Nemesis (Thanks to randerscheinung).

[LordHelmchen]

Library update for Dragon's Maze cardcount.
The two $ sitescripts updated for latest MA version, which means Dragon's Maze is included.
The € sitescripts will follow once the vendors have updated their shops; those will also have updated expected counts for German MM and PR. Until then, if you use both, you'll need to keep both library versions and don't worry about unexpected non-fails in the two sets.

[LordHelmchen]

fixed a small problem with savepath handling in LHpi-v2.2.lua. if you downloaded already, please redownload the fixed version.

[LordHelmchen]

Release Day is upon us and the German shops have updated their sites. So here's the other two sitescripts, with Dragon's Maze and up-to-date expectedcounts. mtgmintcards received a very minor update (two sets have a new expected drop count), but unless you turn VERBOSE to true, you wouldn't even notice those.

[LordHelmchen]

Library update for MMA (cardcount) and M14 (cardcount and basic lands variants)
updated all four sitescripts to make use of the new version and import MMA and M14 (no MMA from magicuniverse). Note that some shops may not have yet updated their sites, so you might not see new prices before release day.

As usual, for documentation read LHpi's Wiki page; post any questions and/or praise in this thread

[LordHelmchen]

Library update for Theros (cardcount and basic lands variants).
Only tcgplayer PriceGuide yet. The German sites are not updated yet (and LHpi.trader-onlineDE needs additional work to funtion again).
LHpi-mtgmintcards broke as well (read this for details) and might be discontinued (I'm not sure yet whether it's worth the work).

[LordHelmchen]

Ok, adapted for multiple pages per set/rarity/foilage to work with the new mtgmintcards limit of 200 results per page. There's still some calls for empty pages, but I think those are worth having a working import again. (I'm looking forward to seeing Goblin Hero's solution.)
The German sites are only preemptively Theros-readied, as once again the shops won't update until releaseday. But at least all four scripts are working with the changed BuildUrl/GetSourceData functions.

Wiki page is still there for documentation, and I'm still open to any suggestions or questions you might have.

[LordHelmchen]

Site was updated, and so is the script.
And since I changed some functions in the library, the empty template got adapted as well.

[LordHelmchen]

and here's the fourth one.

[LordHelmchen]

As usual, see the LHpi wiki page for instructions and changelog.

In short, Born of the Gods and a few special sets (Duel Decks, Un-sets) were added.

Goblin Hero: I have problems setting “Ach! Hans, Run!” (UNH), Kongming, “Sleeping Dragon” and Pang Tong, “Young Phoenix” (both PTK). Can you tell me which character (and to be safe, which byte sequence) MA expects for the quotation marks?

Also, Commander2013 showed that I don't know how I can distinguish regular and oversized versions of the same card for setting the price. Is there a simple and obvious way to do it? If there is no way at all yet, how about using the version field to mark oversized versions?