Upscale low res card images to high res using Deep Learning

[austinio7116]

All done.

PM me if you are interested

[jenjia]

Just my 2 cents,

There are programs/websites where you can make "fake" cards, just use those for the text/borders/etc and use your algorithm to generate only the image.

In the end, if will be much better

[austinio7116]

True, but that would take ages. The computer does all the work here.

[nesci]

Really good job and interesting posts.

I think @jenjia has a point and I also think there's a way to make the computer do "almost" all the work.

1) Create all the cards without the artwork with @Ahkren's cards generator.
2) Use ImageMagick to extract the artwork of your rendered cards
3) Use ImageMagick to put the artwork on the cards from the generator

What do you think?

[austinio7116]

Thanks - I'll take a look at Ahkren's generator - looks good. I guess the only fiddly bit would be scripting imagemagick differently for each card type (full art lands, aftermath, planeswalkers etc).

[austinio7116]

Also - the upsampling effect on the art will never be as good as the borders as each image is different - so I'm not sure how much it would improve the generator results to use these upsampled images - it does seem to learn a good sharpening model though so perhaps it would be better than nothing.

[austinio7116]

Got the generator running - just need to create some masks for imagemagick and I'll be ready to try out your suggestions. Still not sure quite how the planeswalker art is going to work though.

[austinio7116]

I may be able to get high res art for all the planeswalkers online now I suppose

[austinio7116]

Another example of the final superresolution effect - I think it does a good enough job on the art to make it worth using in combination with the HQ card generator:

These images show first the low res image, then bicubic upsampling - then the neural network output:

[austinio7116]

Here is how the HQ card generator border looks in comparison:

Clearly this is much better in terms of the text, however I would argue that the upscaled image from the network is closer to the real look of the card. I may continue to use the upscaled images (as I only use 150dpi on Android forge - and at that resolution the upscaled images are pretty much as good as downsampled scans. It depends what you want from the cards though - I still may try to script up some imagemagick to see how the whole set looks with the the HQ borders and text. I can also try increasing the amount of training data and the size of the network to see if I can get the super resolution any better.

[jenjia]

You should try to train your deep learn network with only the artwork (low & high res), when using the HQ Card Generator.

I think the borders are just noise for the network. Just a guess.

(Disclaimer: I also work with Computer Vision, if you want you can check my CCG Scanners for iOS and Android, named: MTG Manager, Card Scanner for YuGiOh, Card Scanner for PokeTCG, Card Scanner for Vanguard )

[austinio7116]

Yes, I would have to retrain on just the art to get the best results.

[nesci]

How do you train your deep learning network?
You got me interested in your project and I have a few very important pictures that are low quality that I would be more than happy to enlarge their resolution.

[austinio7116]

I used the code provided here as a starting point - just tweaking some parameters and a few changes to the model to better work for this problem (I removed the batch normalization layers from the generative model to avoid different parts of the final card having different intensity profiles):

https://github.com/david-gpu/srez

That network is designed to upsample a 16x16 square to 64x64 by training on high res image squares (which it downsamples 4x to use as input). I wrote some python tensorflow code to use the trained model to upsample new 16x16 inputs (as the provided code did not include this part) then wrote some python to loop over a low res spoiler HOU card image, chopping it into 16x16 blocks every 4 pixels (so the blocks overlap). These 16x16 blocks are then upsampled by the network in batches (to avoid running out of memory on the graphics card - all this works with tensorflow using CUDA for best performance). The 64x64 blocks are then patched together into the high res card averaging the overlapping blocks for best results.

The training data I used in the latest version is the AKH high res survey images with a bit of de-noising applied in Photoshop to remove JPEG noise and artifacts. These images are chopped up into 64x64 blocks (100,000+) by looping over the card in increments of 16 pixels.

If you want to use these kinds of networks to upsample images you need to have high res examples of similar images to train the network on - if your training images don't contain similar features to the images you want to upsample you will not get good results.

Training takes around 6 hours (before my graphics card runs out of memory - I'd like to train it longer) - then each card takes 3 minutes to upsample.

[austinio7116]

I have increased the size of the network layers slightly in the hope of more detailed upsampling - I am processing the full set again as I think this is a (very) small improvement over the previous versions:

The artist info text is clearer, the text is very slightly sharper and better representative of the original font and the edition icon and mana icons are better.

I may process this version on the full set, then I will look to increase the amount of training data and try to find a way to get the network to train for longer without running out of memory.