Upscale low res card images to high res using Deep Learning

[skibulk]

This is an interesting project. Have you dabbled with OCR at all? I think you might get better results if you use machine learning to re-construct the card without the text and then layer the text back in using a real font. You can use OCR to detect positioning and line breaks, to calculate leading. Microsoft's OCR service is pretty awesome: https://azure.microsoft.com/en-us/servi ... er-vision/

I'd suggest using the HQ Card Generator instead, but you'll always encounter a delay in the release of new frames and symbols. It does have all the fonts though. You can find that software here: viewtopic.php?f=30&t=17795

[austinio7116]

Yes, I had thought of doing that, but I'm pretty new to neural networks and the network I have is for super resolution. If I had the time to learn how to build the network you suggest (which I presume would need some custom layers) that would be the way to get very good results at least on the text. I think this super resolution method should get pretty good results if I can optimize the training process a bit more. If anyone else has the skills and time to try integrating OCR too that would be great to see though.

[austinio7116]

Some examples of what version 1 will look like - processed 14 hours so far:

[austinio7116]

This is an interesting project. Have you dabbled with OCR at all? I think you might get better results if you use machine learning to re-construct the card without the text and then layer the text back in using a real font. You can use OCR to detect positioning and line breaks, to calculate leading. Microsoft's OCR service is pretty awesome: https://azure.microsoft.com/en-us/servi ... er-vision/

I'd suggest using the HQ Card Generator instead, but you'll always encounter a delay in the release of new frames and symbols. It does have all the fonts though. You can find that software here: viewtopic.php?f=30&t=17795

One example where the pure superresolution approach may actually be better would be something like the "Hour of" series - where the text is overlayed on a different textured background - of course OCR would still work if you had enough background templates to cover each scenario but the generic approach may be more flexible to future changes in templates:

[Agetian]

Looking very good so far! Keep up the great work!

- Agetian

[austinio7116]

Quite pleased to see that in some cases the art is upscaled quite unexpectedly well:

Low res

Upscaled

[austinio7116]

Finished the first pass of processing the full HOU set. There are a few missing cards that I need to add and a bit of post processing left to do before I have something ready to share.

[austinio7116]

First full set of HOU at upsampled 200dpi is complete and post processed - should be available somewhere soon:

[austinio7116]

I overlaid random foils from AKH on the foils

[austinio7116]

Some animated GIFs showing the upsampling effect of the network:

[austinio7116]

I think I've worked out why the survey images training set didn't work as well. If you zoom in on the high res survey jpgs there is a lot of noise around the artist/collection no text at the bottom of the card (and the rest of the text too) - I suspect that the network is learning to replicate that noise. I will try pre-filtering the training set to remove the noise and the JPEG artifacts and re-training on that.

[austinio7116]

I think I've a good theory as to why the left and right borders are overexposed. The network works on small patches, so to keep the calculation within memory on my graphics card I am batching the upscaling in columns. The network contains batch normalisation layers which are therefore batch normalizing the columns containing mostly black border at the edges which is different from the random batches it sees during training. I can try removing these layers from the model, or failing that restructure the batching process sonthe black border patches are distributed evenly.

[austinio7116]

I think I've a good theory as to why the left and right borders are overexposed. The network works on small patches, so to keep the calculation within memory on my graphics card I am batching the upscaling in columns. The network contains batch normalisation layers which are therefore batch normalizing the columns containing mostly black border at the edges which is different from the random batches it sees during training. I can try removing these layers from the model, or failing that restructure the batching process sonthe black border patches are distributed evenly.

Removing the normalization layers from the network has fixed all of the intensity/colour issues that were present in the first batch - especially the over-exposure of the borders. I just need to find some ideal parameters for training on the survey data and I will be able to generate a much prettier second batch of HOU images.

[austinio7116]

Here's an example of a raw output of the new network with the normalization issues fixed. I hope to find a better set of training parameters to improve the resolution further before running again on the full set:

[austinio7116]

I have now found some parameters that I am happy with and am processing the full set. Here are a few raw examples:

It only takes 3 minutes per card now so the full set should not take long.