Gatherer Downloader App (v1.6.8) [hq pics / card database]
Moderator: CCGHQ Admins
Re: Gatherer Downloader App (v1.1.0)
by woogerboy21 » 28 Feb 2011, 18:55
Well my original idea about having the ability to define paths in the set names field was so you didn't have to write any customization per supported application into your applicaton (gatherer downloader app) to accomidate supported application folder structures [confused? lol]. The ability to define custom folder names exists already by placing the custom folder names after the "/" in the set list line so taking it one step further to create a complete path didn't seem to much of a leap. Which by the way did you actually work that in? (sorry lost track of that in the various respones) If that was there the end user could simply add the information in if they wanted (or others could offer some type of txt file listing that could easily replace whats in there existing sets list with each application specific folder structure definition). That way your application itself still retains its original functionality and any type of folder structuring can be put in place by the end users (if needed). And of course the default button returning the sets list back to the default settings that you have defined so if they ever muck something up they can fix it easily.
Though I like the idea of changing the functionality of the defaults button to a drop down list for supported applications I wouldn't want to maintain a listing of each supported applications image folder structure per release. If you want to great! (not I) I would just leave the set's listing screen like it is and adding the ability for your application to create the folder structure defined. Then possibly widen the screen to accomidate the larger lines that may be put in place for folder structures (nothing to huge though
).
Thoughts?
edit:
btw 1.1.0 is working great so far!
Though I like the idea of changing the functionality of the defaults button to a drop down list for supported applications I wouldn't want to maintain a listing of each supported applications image folder structure per release. If you want to great! (not I) I would just leave the set's listing screen like it is and adding the ability for your application to create the folder structure defined. Then possibly widen the screen to accomidate the larger lines that may be put in place for folder structures (nothing to huge though
).Thoughts?
- setinfo | Open
edit:
btw 1.1.0 is working great so far!
-
woogerboy21 - HQ Team Member
- Posts: 1136
- Joined: 19 Jul 2009, 00:15
- Location: USA
- Has thanked: 21 times
- Been thanked: 152 times
Re: Gatherer Downloader App (v1.1.0)
by Ark » 28 Feb 2011, 20:19
The {variable} idea I felt would actually be way more complicated than need be for most users, and as far as I know, I have most of the formatting options the two major programs I know about would need, now with v1.1.0.
I suppose you can use the alternate set code field as a full set name if you want, but then the set name cannot contain a "-" character, because of how I parse the lines. While this alternate set code field does benefit Magic Workstation for some of the older sets, the real reason I *had* to implement it is because of Conflux. CON is a reserved file name which refers to the "console" with the command prompt. So even though CON is the official set code for Conflux, MWS is forced to use a folder called CFX, because it is absolutely impossible to create a folder called CON under Windows.
Perhaps I can extend the set list format to include 4 columns so you can have Official Set Code, Set Code for Saving, Official Set Name, Set Name for Saving. Then I can include an option to use the Official Set Name or the Set Name for Saving, and Magic Album can use that field. I need to come up with a better way to parse the lines then, perhaps CSV?
As an example:
CON,CFX,Conflux,Expansions\Conflux
TSB,,"Time Spiral ""Timeshifted""",Expansions\Time Spiral Timeshifted
M11,,Magic 2011,Core Sets\Magic 2011
PD2,,Premium Deck Series: Fire and Lightning,Promos\Premium Deck Series - Fire and Lightning
I would need your help to ensure the alternate paths for each set are useful for Magic Album so that my defaults are correct.
Btw the window for the set editor is resizable currently, even if the default size is small.
I suppose you can use the alternate set code field as a full set name if you want, but then the set name cannot contain a "-" character, because of how I parse the lines. While this alternate set code field does benefit Magic Workstation for some of the older sets, the real reason I *had* to implement it is because of Conflux. CON is a reserved file name which refers to the "console" with the command prompt. So even though CON is the official set code for Conflux, MWS is forced to use a folder called CFX, because it is absolutely impossible to create a folder called CON under Windows.
Perhaps I can extend the set list format to include 4 columns so you can have Official Set Code, Set Code for Saving, Official Set Name, Set Name for Saving. Then I can include an option to use the Official Set Name or the Set Name for Saving, and Magic Album can use that field. I need to come up with a better way to parse the lines then, perhaps CSV?
As an example:
CON,CFX,Conflux,Expansions\Conflux
TSB,,"Time Spiral ""Timeshifted""",Expansions\Time Spiral Timeshifted
M11,,Magic 2011,Core Sets\Magic 2011
PD2,,Premium Deck Series: Fire and Lightning,Promos\Premium Deck Series - Fire and Lightning
I would need your help to ensure the alternate paths for each set are useful for Magic Album so that my defaults are correct.
Btw the window for the set editor is resizable currently, even if the default size is small.
Re: Gatherer Downloader App (v1.1.0)
by Ark » 28 Feb 2011, 20:25
Actually, because of how paths are created, and to save on duplicate confusing entries, I think not allowing \ in the fields is easier. It's not impossible to do, but this may be a more simply method both for me to program and for you to list things:
CON,CFX,Conflux,Expansions
TSB,,"Time Spiral ""Timeshifted""",Expansions
M11,,Magic 2011,Core Sets
PD2,,Premium Deck Series: Fire and Lightning,Promos,Premium Deck Series - Fire and Lightning
The 5th column in this case would be equal to the 3rd column if blank, and the 4th column would just be for a per-category subfolder name. Similarly, the 2nd column is assumed equal to the 1st if left blank.
CON,CFX,Conflux,Expansions
TSB,,"Time Spiral ""Timeshifted""",Expansions
M11,,Magic 2011,Core Sets
PD2,,Premium Deck Series: Fire and Lightning,Promos,Premium Deck Series - Fire and Lightning
The 5th column in this case would be equal to the 3rd column if blank, and the 4th column would just be for a per-category subfolder name. Similarly, the 2nd column is assumed equal to the 1st if left blank.
Re: Gatherer Downloader App (v1.1.0)
by woogerboy21 » 28 Feb 2011, 21:04
Keep in mind everything I'm throwing out is just my $0.02 and its your applicable but I personally find having a field that defines its folder structure entirely without having to determin which field is appended or not appended is easier myself. Also I seem to recall certain applications (such as wagic if memory serves me right) having a folder structure that doesn't even contain the set name. Folder structure might be "\images\1\" or "\images\5" for example and using your example above this type of structure would not be possible to produce using the appending method explained.Ark wrote:Actually, because of how paths are created, and to save on duplicate confusing entries, I think not allowing \ in the fields is easier. It's not impossible to do, but this may be a more simply method both for me to program and for you to list things:
CON,CFX,Conflux,Expansions
TSB,,"Time Spiral ""Timeshifted""",Expansions
M11,,Magic 2011,Core Sets
PD2,,Premium Deck Series: Fire and Lightning,Promos,Premium Deck Series - Fire and Lightning
The 5th column in this case would be equal to the 3rd column if blank, and the 4th column would just be for a per-category subfolder name. Similarly, the 2nd column is assumed equal to the 1st if left blank.
-
woogerboy21 - HQ Team Member
- Posts: 1136
- Joined: 19 Jul 2009, 00:15
- Location: USA
- Has thanked: 21 times
- Been thanked: 152 times
Re: Gatherer Downloader App (v1.1.0)
by Ark » 28 Feb 2011, 21:14
What determines the "1" vs "5" in your example for Wagic? You can use the Multiverse ID for the filename, and disable all subfolders so all files are in one big folder. I thought some application (maybe Wagic?) needed that format.
If I did make a feature with {variables} for the path, there would need to be a variable that defines when to use "1" or "5" and how do I know when to use which? Otherwise if you are just hardcoding one path name for ALL pictures, then just select that path when you click on the button to save the images, it would be the same thing.
Worse comes to worst, if some sets need to go in "1" vs "5" a user could (in this theoretical future version) just change the category column to 1 or 5 as needed I think.
I think this 5-column CSV format would be an elegant solution that allows easier functioning with Magic Album. If somebody who uses Wagic wants to contribute the info about the files/paths needed, I can include Wagic in the dropdown for Load Defaults in a future version.
If I did make a feature with {variables} for the path, there would need to be a variable that defines when to use "1" or "5" and how do I know when to use which? Otherwise if you are just hardcoding one path name for ALL pictures, then just select that path when you click on the button to save the images, it would be the same thing.
Worse comes to worst, if some sets need to go in "1" vs "5" a user could (in this theoretical future version) just change the category column to 1 or 5 as needed I think.
I think this 5-column CSV format would be an elegant solution that allows easier functioning with Magic Album. If somebody who uses Wagic wants to contribute the info about the files/paths needed, I can include Wagic in the dropdown for Load Defaults in a future version.
Re: Gatherer Downloader App (v1.1.0)
by woogerboy21 » 28 Feb 2011, 21:56
My example given simply illustrates that there may be scenario's that alternate foldering structures could exist outside the limitations given in your catagory column solution, not necessarily specific to a given application. Who's to say there is only a single subfolder depth structure or a 5 subfolder depth structure in a given application. Magic Workstion uses a simply single level foldering structure such as \MBS\, Magic Album is a bit deeper \Expansions\Mirrodin Besieged\ but who's to say it couldn't be something like this: \myimages\large\mbs\a\1\{cardname}.jpgArk wrote:What determines the "1" vs "5" in your example for Wagic? You can use the Multiverse ID for the filename, and disable all subfolders so all files are in one big folder. I thought some application (maybe Wagic?) needed that format.
If I did make a feature with {variables} for the path, there would need to be a variable that defines when to use "1" or "5" and how do I know when to use which? Otherwise if you are just hardcoding one path name for ALL pictures, then just select that path when you click on the button to save the images, it would be the same thing.
Worse comes to worst, if some sets need to go in "1" vs "5" a user could (in this theoretical future version) just change the category column to 1 or 5 as needed I think.
I think this 5-column CSV format would be an elegant solution that allows easier functioning with Magic Album. If somebody who uses Wagic wants to contribute the info about the files/paths needed, I can include Wagic in the dropdown for Load Defaults in a future version.
Creating a routine that simply takes the users input and creates the folder structure defined by the user and places the pictures with in the users defined folder structure sounds like a much simpler solution than trying to account for each and every application ever created inside your compiled code. Not to mention its adds the flexibility for later on supporting newly created application image foldering structures without having to be an update requested.
I was just trying to work in a solution that would require the least future maintenance to the downloader application itself, that is all.
-
woogerboy21 - HQ Team Member
- Posts: 1136
- Joined: 19 Jul 2009, 00:15
- Location: USA
- Has thanked: 21 times
- Been thanked: 152 times
Re: Gatherer Downloader App (v1.1.0)
by Ark » 28 Feb 2011, 22:07
Maybe I'm not getting what you're trying to say, but if the you entered \myimages\large\mbs\a\1\{cardname}.jpg into a box, and only {cardname} is a variable, then the rest is really irrelevant, if it is the same for every single card.
Just go create \myimages\large\mbs\a\1\ yourself manually, and select that folder to create all the card images in.
Just go create \myimages\large\mbs\a\1\ yourself manually, and select that folder to create all the card images in.
Re: Gatherer Downloader App (v1.1.0)
by woogerboy21 » 01 Mar 2011, 01:32
I apologize if I haven't been clear. I was refering to the path as being per set. Each set could have a different entry in its line for a different location. So for alpha i may have a path of \myimages\large\alpha\ and for beta it may be \yourimages\small\dumbdumbdumb\ for that matter (and so on for each set if need be). Each sets folder structure is the variable and the {cardname}.jpg would be the name of the card that is downloaded from the gatherers website (whatever card that may be.) There would be a high likely-hood that most sets will fall into similar paths such as the magic album application having the base "core sets", "expansions", "promo cards" etc but allowing for a definition of folder structure per set would give the downloader application the flexibility for future third party applications changes as well as newer third party application support. Any clearer?Ark wrote:Maybe I'm not getting what you're trying to say, but if the you entered \myimages\large\mbs\a\1\{cardname}.jpg into a box, and only {cardname} is a variable, then the rest is really irrelevant, if it is the same for every single card.
Just go create \myimages\large\mbs\a\1\ yourself manually, and select that folder to create all the card images in.
- example | Open
edit:
I understand that the folders will be created in whatever location you choose to download the images to but what I am trying to show in the example is that with the flexibility of being able to produce folder structures deeper than a single subfolder depth per set will allow more flexibility for third party application support without having to hard code in special cases per application you wanted to support.
-
woogerboy21 - HQ Team Member
- Posts: 1136
- Joined: 19 Jul 2009, 00:15
- Location: USA
- Has thanked: 21 times
- Been thanked: 152 times
Re: Gatherer Downloader App (v1.1.0)
by Ark » 01 Mar 2011, 01:53
I see the argument for allowing multiple folders in the set list, but I don't see how it comes to any need for a complex {variable} system.
I'm still leaning towards the 5-column CSV format: GathererCode, SaveCode, GathererName, Category, SaveName
SaveX should be blank if it is equal to GathererX, and Category could potentially contain mutliple subfolders, although for now, being a single folder name gets everything you need for Magic Album.
I'm still leaning towards the 5-column CSV format: GathererCode, SaveCode, GathererName, Category, SaveName
SaveX should be blank if it is equal to GathererX, and Category could potentially contain mutliple subfolders, although for now, being a single folder name gets everything you need for Magic Album.
Re: Gatherer Downloader App (v1.1.0)
by woogerboy21 » 01 Mar 2011, 01:56
Maybe once again I haven't been clear. There is no need for a variable system (simple or complex). The 5 column csv is a good solution, all I was asking for is the ability to create a foldering structure greater than a single subfolder depthArk wrote:I see the argument for allowing multiple folders in the set list, but I don't see how it comes to any need for a complex {variable} system.
I'm still leaning towards the 5-column CSV format: GathererCode, SaveCode, GathererName, Category, SaveName
SaveX should be blank if it is equal to GathererX, and Category could potentially contain mutliple subfolders, although for now, being a single folder name gets everything you need for Magic Album.
Magic Album is a great application I use for inventory purposes but there are other uses for card images and having a method for making the folder structure built into the downloader application would come in handy thats all.
-
woogerboy21 - HQ Team Member
- Posts: 1136
- Joined: 19 Jul 2009, 00:15
- Location: USA
- Has thanked: 21 times
- Been thanked: 152 times
Re: Gatherer Downloader App (v1.1.0)
by woogerboy21 » 01 Mar 2011, 18:26
One other thing I have noticed with the most recent release (1.1.0) that I don't remember seeing in the older version is if you start to download all sets after downloading for a period of time the application begins to not respond according to windows. It still is downloading images if you look in the directory you told it to download, but after it completes getting all the images you are unable to use the UI. Have you seen this?
It also appears the images get cropped and in doing so if you attempt to download sets such as "archenemy" that contain scheme cards, the scheme cards are cropped improperly.
It also appears the images get cropped and in doing so if you attempt to download sets such as "archenemy" that contain scheme cards, the scheme cards are cropped improperly.
-
woogerboy21 - HQ Team Member
- Posts: 1136
- Joined: 19 Jul 2009, 00:15
- Location: USA
- Has thanked: 21 times
- Been thanked: 152 times
Re: Gatherer Downloader App (v1.1.0)
by Ark » 01 Mar 2011, 19:34
I have not seen the UI stop responding. What OS are you using? The downloading happens in a separate thread spawned just to do the downloading, so that the main UI can continue as normal. This allows you to be able to click the Cancel button, so they really don't interfere with each other. I've only really tested under Windows 7 Professional 64-bit on a Quad Core i7, but I can't see any reason it would act differently on other OSes or single core CPUs.
About crops, I stated when I added the "&special=true" flag that non-standard cards are not going to work with Fulls or Crops, because the size is assumed to be the regular card size. It is also not quite perfect with pre-8th edition cards on the Crops as the art is slightly different sized. I might be able to change this to look at the image size and use different numbers depending on the card size, but I can't really fix the pre-8th edition Crops because the base card size is the same.
About crops, I stated when I added the "&special=true" flag that non-standard cards are not going to work with Fulls or Crops, because the size is assumed to be the regular card size. It is also not quite perfect with pre-8th edition cards on the Crops as the art is slightly different sized. I might be able to change this to look at the image size and use different numbers depending on the card size, but I can't really fix the pre-8th edition Crops because the base card size is the same.
Re: Gatherer Downloader App (v1.1.0)
by woogerboy21 » 01 Mar 2011, 20:40
Im running Windows 7 (various versions). I have tried it on 64-bit professional, 32bit professional and 32bit home premium. Two systems were dual core, and a third was dual core hyper threaded but I don't see how the number of processors would affect the behaviour really. They all exhibit the same behaviour. Other things to take into consideration when the downloader is running is I do have it behind other windows so loosing and gaining focus might have something to do with it. Also I would have expected after the downloading completes that the UI to start responding again but it does not and I have to terminate it using task manager.Ark wrote:I have not seen the UI stop responding. What OS are you using? The downloading happens in a separate thread spawned just to do the downloading, so that the main UI can continue as normal. This allows you to be able to click the Cancel button, so they really don't interfere with each other. I've only really tested under Windows 7 Professional 64-bit on a Quad Core i7, but I can't see any reason it would act differently on other OSes or single core CPUs.
About crops, I stated when I added the "&special=true" flag that non-standard cards are not going to work with Fulls or Crops, because the size is assumed to be the regular card size. It is also not quite perfect with pre-8th edition cards on the Crops as the art is slightly different sized. I might be able to change this to look at the image size and use different numbers depending on the card size, but I can't really fix the pre-8th edition Crops because the base card size is the same.
-
woogerboy21 - HQ Team Member
- Posts: 1136
- Joined: 19 Jul 2009, 00:15
- Location: USA
- Has thanked: 21 times
- Been thanked: 152 times
Re: Gatherer Downloader App (v1.1.0)
by Ark » 01 Mar 2011, 20:49
Can you tell me all options you have set (set list, language/additional information, image formats screen, "and all sets after") so I can exactly reproduce the set of things you are downloading?
Did this happen in previous versions, or only in v1.1.0?
At what point does the UI stop responding? How many seconds/minutes in?
Can you reproduce the issue with a certain number of alt-tab or specific application switches? Does it matter what other applications are running?
The downloader thread, other than setting a string which the main GUI will display on the status line, does not really interact with the GUI, and once the download is complete, it simply terminates and goes away. So if the GUI thread freezes for some reason, there is no reason the downloader thread would wake it up when it exist.
Did this happen in previous versions, or only in v1.1.0?
At what point does the UI stop responding? How many seconds/minutes in?
Can you reproduce the issue with a certain number of alt-tab or specific application switches? Does it matter what other applications are running?
The downloader thread, other than setting a string which the main GUI will display on the status line, does not really interact with the GUI, and once the download is complete, it simply terminates and goes away. So if the GUI thread freezes for some reason, there is no reason the downloader thread would wake it up when it exist.
Re: Gatherer Downloader App (v1.1.0)
by woogerboy21 » 01 Mar 2011, 21:33
Let me re-run it again to night and get you some time frames. I can tell you the configuration is the sets list I posted above in containing all the sets from the gatherer. I start at Limited Edition Alpha and download all sets afterwords. I then go into the options->image formats window and select the magic album option from the drop down list and choose load defaults. Then choose download->card images selecting my desktop to download the images to. The application creates a folder labeled "Fulls" on my desktop and begins downloading the images. I have tried it multiple times trying to determin were it actually causes the UI to stop responding but have been unable to determin if there is a specific set or time period it occurs in. I have tried it with other windows open and it just running on its own and the UI always begins to not respond after a good length of time downloading the card images. I can tell its still downloading images though by two methods, one method is looking at the properties of the "Fulls" folder and you can see the file count increase and also along the windows start bar in 7 it shows the smaller icon of the running application fill with the neon greenish color and once completes goes back to the blue background. How ever the UI itself after it stops responding will never respond again even after the downloading completes. At first I thought it was possibly some of the newer sets that I posted and maybe one of them causing a problem but even if I use the applications default set's list it happens. To me it seems like there is some type of expected application UI update timeout that isn't occuring and windows is marking the application UI thread as not responsive (but this is just a guess). If that is the case and you can reproduce the behaviour you might be able to fix it with some kind of action between sets. If you do just a few sets I don't get the UI non-responsive issue, so it definatly has to do with the quantity of set images I am downloading.Ark wrote:Can you tell me all options you have set (set list, language/additional information, image formats screen, "and all sets after") so I can exactly reproduce the set of things you are downloading?
Did this happen in previous versions, or only in v1.1.0?
At what point does the UI stop responding? How many seconds/minutes in?
Can you reproduce the issue with a certain number of alt-tab or specific application switches? Does it matter what other applications are running?
The downloader thread, other than setting a string which the main GUI will display on the status line, does not really interact with the GUI, and once the download is complete, it simply terminates and goes away. So if the GUI thread freezes for some reason, there is no reason the downloader thread would wake it up when it exist.
Oh and another thing I noticed, if after downloading the database information with all the sets selected and saving it to an xml file. If I reopen the application and choose to load that large of an xml file the application goes un-responsive till the complete xml file is loaded then begins to respond again.
-
woogerboy21 - HQ Team Member
- Posts: 1136
- Joined: 19 Jul 2009, 00:15
- Location: USA
- Has thanked: 21 times
- Been thanked: 152 times
Return to Magic Rules Engine Programming
Who is online
Users browsing this forum: No registered users and 16 guests