How-To sitescriptTemplate

Introduction

This wiki page give information about properties and function you can find in LHpi sitescriptTemplate file. It helps you to understand what you must/can define and how-to define it.

Any remarks or feedbacks are welcome.

Properties

LHpi library version

Make sure that your script defines the version of LHpi library it uses.

The property to adapt in sitescript file is: libver

Example:

libver = "2.7"

Name of sitescript

The first thing to do is to define the sitescript filename in order for LHpi library to know which one is executed.
It is needed for example if you want to save your HTML page locally for later offline process.

The property to adapt in sitescript file is: scriptname

Example:

scriptname = "LHpi.TheWebsiteName-v".. libver .. ".1.lua"

It is a good choice to leave the library version into the filename in order to know, without editing sitescript file, which version of LHpi library is used.

Regular expression to retrieve card info

Next thing to configure is the regular expression used by LHpi to retrieve all information of cards from HTML page.
You can find useful information about LUA regular expression in the LUA reference page: [1]

The regular expression should match all cards into the HTML page, where each entry of the result correspond to ONE card information.
It is also important that the regular expression defines ONE GROUP pattern including the name and price of the card.
When available, GROUP pattern could also include the FOIL flag when both regular and foil prices are available on the same HTML page.

To test your regular expression, you can download locally (with your browser) from your website the HTML page of a set.
Edit that HTML page with Notepad++ and use the Find function (CTRL + F). In the Find windows, on the bottom left in Find Mode, you have the choice "Regular Expression".
Select that option and try your regular expression. Each time you click on Next button, the selection done by Notepad++ should highlight all information of ONE card, with a different card highlighted at each Next click.

The property to adapt in sitescript file is: site.regex

Example: site.regex = "(<div.-name=\".-\".-foil=\"%d\".-class=\"price\">.-</span>)"

Currency

You can also define currency used by your website when different from the dollar '$'.

The property to adapt in sitescript file is: site.currency

Example: site.currency="€"

(Optional) Regular expression about expected results

If the website displayed information about the set, for example the number of card, you can define the regular expression to retrieve these informations.
LHpi will use it in order to check that the import of card data matches the expected results returned by this regular expression.

--More information aprreciated--

The property to adapt in sitescript file is: site.resultregex

By default, property is not set.
When property is not defined, the expectation is not verified.

(Optional) Site Encoding

When the website used a specific encoding (e.g. UTF-8), you can define it in this property.
A correct encoding definition is needed in order for LHpi to process card data correctly and avoid unexpected behavior.

The property to adapt in sitescript file is: site.encoding

Example: site.encoding="UTF8"
By default, the property is not set and default encoding used is "cp1252"

Site languages

The language of cards supported by the website should be defined here.

By default, English language is defined but you can add more languages when supported.

The property to adapt in sitescript file is: site.langs

Example:

 site.langs = {
      [1] = {id=1, full = "English", abbr="ENG",  url="eng" },
      [4] = {id=4, full = "French",  abbr="FRA",  url="fra" }
 }

The format of each entry is:

[<langid>] = {id=<langid>, full = "<Fullname>", abbr="<AbbrName", url="<ParameterValue>"]

where

• <langid> is the ID of language as defined by Magic Album in file \Database\Languages.txt
• <Fullname> is the full name of the language
• <AbbrName> if the abbreviated name of the language
• <ParameterValue> is the value that can be appended to the website url when retrieving cards in that language

FRUC - Foil Rare Uncommon Common table

The property defines all type of card available on the website.
This table is useful when card information are available at different urls depending of them rarity/foil.

The property to adapt in sitescript file is: site.frucs

Example: site.frucs = { "foil", "regular" }

Site sets

The most important property of the sitescript.

The property to adapt in sitescript file is: site.sets

It defines the mapping between the Magic Album (MA) and the website supported sets.
It says to LHpi library that the MA set with id xxx is available on the website with foil/regular prices, in which language, at specific url suffix.

Here is the format of one set mapping:

[<setid>]={id = <setid>, lang = { <eng avail>, [<langid>]=<lang avail>}, fruc = { <foil avail>, <regular avail> }, url = "<url suffix>"},

where

• <setid> is the id of the set as defined by MA. See file \Database\Sets.txt
• <eng avail> is a boolean indicating whether the price information is available for cards in ENGLISH language or not
• <langid> is the id of a language as defined by MA. See file \Database\Languages.txt
• <lang avail> is a boolean indicating if the price information is available for cards in language defined by <langid> (previous marker)

NOTE: <langid> and <lang avail> markers are used together when website supports card information in language different than English. If no other language is available, only <eng avail> marker is set.

• <fruc avail> is a boolean indicating whether price of foil cards is available or not
• <regular avail> is a boolean indicating whether price of regular cards is available or not.

NOTE: You can have more boolean in the fruc array. The number of boolean depends on the FRUC property you defined previously. If the (previous) property has an array of 4 fruc (e.g. {"foil", "rare", "uncommon", "common"}) then the fruc property in the set mapping would have 4 boolean: {true, true, true, true} each one defining whether the price of corresponding fruc is available or not.

• <url suffix> is the suffix added to the website url in order to retrieve price of cards for the current set.

Example:

[800]={id = 800, lang = { true, [4]=true}, fruc = { true, true }, url = "1966"}, -- Theros

• 800 is the id of Theros set defined by MA
• Both English and French languages are available on the website for card price
• Both foil and regular prices are available on the website. Assume than site.fruc is defined like {"foil", "regular"}
• The id of the Theros set on website is 1966. That id (=suffix) will be appended to the url when retrieving card prices of Theros.

Card name replacement

Sometimes, the card name given by the website does not match the card name defined in MA.
You need to match both in order to set the price. This is the objective of the property.

The property to adapt in sitescript file is: site.namereplace

The format is:

 {
  [<setid>] = { 
     ["<website cardname 1>"]	= "<MA cardname 1>",
     ["<website cardname 2>"]	= "<MA cardname 2>"
  }
 }

where

• <setid> is the id of the set defined by MA. See file \Database\Sets.txt.
• <website cardname> is the name of the card as defined by the website
• <MA cardname> if the name of the corresponding card defined by MA

Note:

• A set can have more than one namereplacement mapping (cardname 1, cardname 2, ...)
• site.namereplacement can have more then one set replacement definition

Variants

--TO BE verified--

Some sets has variants of the same card, for example basic land.
The variants property defines all of this and does mapping between the name of card given by the website and its variant in MA.

The property to adapt in sitescript file is: site.variants

The format is: { [<setid>] = { ["<website cardname>"] = { "<MA cardname>", { <variants name> } }, ... } ... }

where

• <setid> is the id of the set defined by MA. See file \Database\Sets.txt.
• <website cardname> is the variant name of the card as defined by the website
• <MA cardname> if the name of the corresponding card defined by MA
• <variants name> if the array of variants name as defined by MA. For now, it is0{ 1, 2, 3, 4}

Example:

[800] = { -- Theros
  ["Plains"] 						= { "Plains"	, { 1    , 2    , 3    , 4     } },
  ["Island"] 						= { "Island" 	, { 1    , 2    , 3    , 4     } },
  ["Swamp 1"] 						= { "Swamp"		, { 1    , false, false, false } },
  ["Swamp 2"] 						= { "Swamp"		, { false, 2    , false, false } },
  ["Swamp 3"] 						= { "Swamp"		, { false, false, 3    , false } },
  ["Swamp 4"] 						= { "Swamp"		, { false, false, false, 4     } },
  ["Mountain"] 					= { "Mountain"	, { 1    , 2    , 3    , 4     } },
  ["Forest"] 						= { "Forest" 	, { 1    , 2    , 3    , 4     } }
}

Website uses the same name for Plains, Island, Mountain and Forest variants. So the price for these card will be identical for all variants.
However, website does distinction between variant of Swamp. So "Swamp 1" is mapped to the variant 1 of Swamp in MA, "Swamp 2 is mapped to variant 2, ...

Please note that defining a set variant in sitescript will override all default variants defined by LHpi library for that set. So make sure to define all variants in your sitescript to avoid losing variants previously defined by LHpi.

Foil tweak

TODO Document me !

Expected results

When you know exactly the information that LHpi should retrieve from website, you can define these expectation in this property.

The property to adapt in sitescript file is: site.expected

The format is:

{
  [<setid>] = {  pset={ <nb of card in ENG>, [<langid>]=<nb of card in lang> }, failed={ <nb of failed in ENG>, [<langid>]=<nb of failed in lang> }, 
                 dropped=<nb of dropped card>, namereplaced=<nb of card with name replacement>, foiltweaked=<nb of card foil tweaked> },
}

where

• <setid> is the id of the set defined by MA. See file \Database\Sets.txt.
• <nb of card in ENG> is the total number of card expected for the set, in English language.
• <langid> is the id of a language as defined by MA. See file \Database\Languages.txt.
• <nb of card in lang> is the total umber of card expected for the set, in the corresponding language.
• <nb of failed in ENG> is the total number of card that LHpi failed to import for the set, in English language.
• <nb of failed in lang> is the total number of card that LHpi failed to import for the set, in the corresponding language.
• <nb of dropped card> is the total number of card that LHpi dropped for the set, all languages combined.
• <nb of card with name replacement> is the total number of card that LHpi did name replacement, all languages combined.
• <nb of card foil tweaked> is the total number of card that LHpi did a foiltweak replacement, all languages combined.

Example:

{
  [788] = { pset={ 249+11, nil, 249 }, failed={ 0, nil, 11 }, dropped=0, namereplaced=1, foiltweaked=0 }, -- M2013
}

Please not that expectation occurs only if LHpi property CHECKEXPECTED is set to true.

LHpi properties

You can customize how LHpi works nor reports events during price imports with the help of Global properties.

• VERBOSE

Control the amount of feedback/logging done by LHpi.
Default is true.

• LOGDROPS

Control whether dropped card must be logged or not by LHpi.
Default is false.

• LOGNAMEREPLACE

Control whether name replacement of card must be logged or not by LHpi.
Default is false.

• CHECKEXPECTED

Control whether expected counter matches the counter of import done by LHpi. See Site Expected results property.
Default is true.

• DEBUG

When true, LHpi logs everything and exit in case of error.
Default is false.

• DEBUGSKIPFOUND

While DEBUG, do not log raw html data found by site.regex.
Default is true.

• DEBUGVARIANTS

Enable DEBUG when card variants are encountered by LHpi.
Default is false.

• OFFLINE

Control whether LHpi read source data from local directory ('savepath' property) only or from site url.
Default is false.

• SAVEHTML

Control whether LHpi save a local copy of each source html to 'savepath' when no in OFFLINE mode or not.
When property is activated, the local directory where LHpi save data must be writable.
Default is false.

• SAVELOG

Control whether LHpi logs to separate logfile (true) or in Magic Album.log (false).
Default is true.

• SAVETABLE

Control whether LHpi save prices into a file (in 'savepath') before importing to Magic Album.
Default is false.

• savepath

Name of existing directory, in \Prices folder, where LHpi read (OFFLINE) or write (SAVEHTML) source html data when corresponding properties are activated.
If this property is not set and LHpi is required to use it (OFFLINE or SAVEHTML activated), then the savepath defined by LHpi corresponds to the scriptname without versioning information.

Functions

ImportPrice ( importfoil, importlangs, importsets )

This is the main function, called by Magic Album, to import prices of selected sets for selected languages, in regular nor foil quality.
It is the entry point where LHpi load it's library LUA file and starts the import magic.

Do not change anything in this function and keep it like this.

site.BuildUrl ( setid, langid, frucid, offline )

The purpose of this function is to construct the url where LHpi will retrieve HTML data containing card information about a specific set, language and FRUC.

Parameters of the function are:

• setid: ID of the set for which url is constructed
• langid: ID of the language to import
• frucid: ID of the FRUC to import
• offline: Boolean flag indicating whether the source are retrieved from local file or from internet

The content of the function already presents in sitescript.lua file can be used as an example.
What you should adapt for your website is following properties in the function:

site.domain = 'www.example.com/prices/'

The domain of the website where you find the prices. Without 'http' prefix.

site.file = 'set.php?context=magic'

The name of the file appended to the domain, and containing HTML parameters which do not concern the setid, langid nor fruc.

site.setprefix = "&set="

The HTML parameter defining the ID of the set on the website.
The corresponding value of the parameter is retrieved from "site.sets[<setid>].url" property.

site.langprefix = "&lang="

The HTML parameter defining the ID of the language on the website.
The corresponding value of the parameter is retrieved from "site.langs[<langid>].url" property.

site.frucprefix = "&fruc="

The HTML parameter defining the ID of the FRUC on the website.
The corresponding value of the parameter is retrieved from "site.frucs[<frucid>]" property.

site.suffix = ""

Any url suffix that can be appended to the url.

The next property in the function you can adapt, if for example some previous parameters are not required, is the url.
In the function, you will find a line like:

local url = site.domain .. site.file .. site.setprefix .. site.sets[setid].url .. site.langprefix .. site.langs[langid].url .. site.frucprefix .. site.frucs[frucid]

which use all properties previously defined in the function.
If some of them are not needed by the website, just remove it from the url construction.

The rest of the function should not be adapted, except if you know what you're doing.

site.ParseHtmlData ( foundstring, urldetails )

The purpose of the function is to extract card information (name, price, ...) from each entry found by the site.regex property in the HTML page.

Parameters of the function are:

1. foundstring: The card entry found by site.regex in the source HTML page. This entry should match information of ONE card.

If you know that the HTML page has more than one entry for the same card (e.g. one entry for regular price, one entry for foil price), process only ONE entry at the time and set the correct properties on the 'card' returned by the function.

1. urldetails: Table containing some information about the current url process.

The informations are the following: { foilonly = #boolean , isfile = #boolean , setid = #number, langid = #number, frucid = #number } Properties 'foilonly' and 'isfile' are the one coming from the site.BuildUrl function.

You must first retrieve card information from 'foundstring' with the help of regular expression or other mechanism that you know. For example:

local _start,_end,name = string.find(foundstring, 'cardname=\"(.-)\"' ) local _start,_end,price = string.find( foundstring , 'class="price">([%d.,]+) .-</span>' )

The price retrieved from html source can actually contain decimal separators. In order for LHpi to support different kind of separators ( 1.000,00 vs 1,000.00 ), it is important that the price you return does not contain these. For example, if the price on the page is '12.50' or '12,50' (12 euros/dollars/<anything else> and 50 cents), the price you should return is '1250'. A good way to do that is:

price = string.gsub( price , "[,.]" , "" ) -- Remove all ',' nor '.' characters from the price price = tonumber( price ) -- Convert price String to Number

With that price, LHpi will divide it by 100 in order to retrieve the cent amount.

Once you're done, you can construct the actual newCard returned with:

local newCard = { names = { [urldetails.langid] = name }, price = { [urldetails.langid] = price } [, <any other properties you can found>] }

Note the marker <any other properties you can found>. In the newCard table, you can define some properties in addition to required names and price. These informations can afterward be used in next function site.BCDpluginPre nor site.BCDpluginPost (explained later).

These extra properties must be defined in a table and added to newCard in under 'pluginData' property. Example:

local newCard = { names = { ... }, price = { ... } , pluginData={ propA=ValueA, propB=ValueB ... }] }

In addition to required 'names' and 'price' properties, you can also set specific properties on the card which will be handled by LHpi:

1. name to override property 'names' used for different language. This 'name' property has precedence over 'names' if it is defined and non empty.
2. lang to override the language abbreviation defined in 'site.lang' for the language having a name defined and non empty.
3. drop in order to drop the card and ignore future treatment by LHpi. The import stop there for that card.
4. foil to mark the card, and its price, as being foiled.
5. variant to override the variants defined in 'site.variants' for the setid and cardname.
6. regprice Force the price of the regular card, and avoid defining price according to card.foil nor card.variant
7. foilprice Force the price of the foiled card, and avoid defining price according to card.foil nor card.variant

4. site.BCDpluginPre ( card , setid )

Function for special cases card data manipulation. Ties into LHpi.buildCardData to make changes that are specific to one site and thus don't belong into the library. This Plugin function is called before most of LHpi's BuildCardData proecssing.

Parameters:

1. card The card, resulting from parseHtmlData and initialized by LHpi after the function call, currently processed.
2. setid ID of the set currently processed.

Return: 'card' modified card for futher processing.

An example of such function is to replace characters in all card name by an alternative. For example:

card.name = string.gsub( card.name , "AE" , "Æ") card.name = string.gsub( card.name , "Ae" , "Æ")

5. site.BCDpluginPost( card , setid )

Function for special cases card data manipulation. Ties into LHpi.buildCardData to make changes that are specific to one site and thus don't belong into the library This Plugin function is called after LHpi's BuildCardData processing (and probably not needed).

Parameters:

1. card The card currently processed by LHpi.
2. setid ID of the set currently processed.

Return: 'card' modified card for futher processing.