tutor.cards needs input

[tutorcards]

Hi everyone.

I'm the developer who made the tutor.cards website and the API for visual card search that MTG Familiar android app uses.
I'm thing about releasing a docker image of the API so developers can use visual search out of the box to create their own apps.

If you're not familiar with the service it's basically this: image in -> most likely card info out.

Currently a traditional computer vision methods for fast image search in a large database is used. That equals about 8GB of total memory used. That's a lot for a normal user. Fortunately I'm working on a convolutional neural network based solution that will drastically reduce the memory needed. It's not there yet but soon (Accuracy is still low).

What features would you expect?

[Arcanist]

Hm, I don't particular have any use for such an API at the moment, but I am curious about the implementation. If you are willing to discuss details, I hope you can answer a few questions:

• What is the accuracy of the current version?
• Is it able to distinguish between different versions of a given card based on expansion symbols and/or card text?
• I assume you are doing some kind of hashing on the pictures, and returning the card which minimizes the distance to that hash. Which image resolution do you use for this?
• How do CNN presently compare to that in terms of accuracy?

[tutorcards]

I assume you are doing some kind of hashing on the pictures, and returning the card which minimizes the distance to that hash. Which image resolution do you use for this?

The matching is done by detecting art. I use feature extraction and feature matching. Best matches are then counted and best image is selected. If there is some glare on the art features won't be detected on that part and chances of finding the correct image decrease. All images are resized to 320px considering aspect ratio.

Is it able to distinguish between different versions of a given card based on expansion symbols and/or card text?

Text, no. Which is good. It can detect the correct card without OCR. I look up the text from a json file provided by mtgjson.
I thought of detecting edition symbols but decided against it because of the drawbacks of using features. By doing CNN I could probably do that. What I do is I provide the list of all the images that use the same art.

What is the accuracy of the current version?

I didn't really measured it. I wanted to put the site up ASAP.
It usually takes 60 ms per image to find a best match. I guess I could sacrifice some more processing time for more accuracy.

How do CNN presently compare to that in terms of accuracy?

Some research in the past years has shown it's becoming state of the art or even surpassing traditional image retrieval.

[Arcanist]

The matching is done by detecting art. I use feature extraction and feature matching. Best matches are then counted and best image is selected. If there is some glare on the art features won't be detected on that part and chances of finding the correct image decrease. All images are resized to 320px considering aspect ratio.

OK. Fingerprinting is an old and proven technique, so I thought this is what you went for when you mentioned the memory requirements. What kind of features do you mean exactly? Would you have a concrete example? Are you talking about detecting specific points of interests (e.g. edges) on pictures by applying a mask?

I thought of detecting edition symbols but decided against it because of the drawbacks of using features. By doing CNN I could probably do that. What I do is I provide the list of all the images that use the same art.

Do you already know for which purpose players use your tool? Off the top of my head I would be likely to use it to:

• find the price of a specific card;
• find the latest rules text of a specific card especially if it happens to be foreign.

Determining the correct printing of the card seems especially relevant for the first case. Maybe instead of applying CNN to recognize card art directly (which is something you can already do), you could focus on recognizing the expansion symbol and the collector info at the bottom of the card. The expansion symbol in particular seems like it would be very useful:

• There are not that many of them, so it is a simpler problem than recognizing card art. It would make sense to apply CNN to tackle it first before moving on to the harder problem. You can even simplify the problem further by converting the picture to grayscale before recognition;
• You could use this information as an additional feature for identifying cards, as recognizing the expansion symbol would considerably reduce the search space;

I didn't really measured it. I wanted to put the site up ASAP.
It usually takes 60 ms per image to find a best match. I guess I could sacrifice some more processing time for more accuracy.

You definitely need to measure it. Without that information, you won't be able to figure out whether changes you make to your algorithm make it better or worse.

Some research in the past years has shown it's becoming state of the art or even surpassing traditional image retrieval.

I know what you can do with CNN. I was wondering about the accuracy you are currently achieving with yours. But it's probably a moot question if you don't know the accuracy of your current algorithm.

Anyway, thanks for taking the time to answer my questions.

[tutorcards]

What kind of features do you mean exactly?

Can't seem to post links yet. So just google Feature Matching with FLANN.
I choose this method because of the scale, translation and rotation invariance. That means the card I want to detect can be in any reasonable position or rotation. I tried fingerprinting but the card needs to be in a certain position to work. Those boxes that you need to align in some apps annoy me.

you could focus on recognizing the expansion symbol

What I could do after the initial search is to warp the query image using matched image to a database image position. And because I know which card it is, I know the layout. I could select that region and perform a template matching for all expansions the art of that card exist. It won't work if the quality of the query image is bad, but will still make the best guess.
Thank you for the suggestion.

And thank you for showing interest.

[Arcanist]

Can't seem to post links yet. So just google Feature Matching with FLANN.
I choose this method because of the scale, translation and rotation invariance. That means the card I want to detect can be in any reasonable position or rotation. I tried fingerprinting but the card needs to be in a certain position to work. Those boxes that you need to align in some apps annoy me.

Ah, it makes sense. I thought you were normalizing the image by rotating it, etc. before feeding it to your algorithm. I did not know about FLANN, but then I don't know much about computer vision even though this is something I have always wanted to pick up. I should really start reading more about it. Thanks for the pointer.

What I could do after the initial search is to warp the query image using matched image to a database image position. And because I know which card it is, I know the layout. I could select that region and perform a template matching for all expansions the art of that card exist. It won't work if the quality of the query image is bad, but will still make the best guess.

That's a good idea. Going back to CNN, another idea is to try and solve the problem assuming cards are properly rotated. It is a simpler problem, and if you cannot get CNN to solve it, then I don't see how you can get them to solve your harder version. If you get that to work, then you have a choice: You can either try and apply CNN to the original problem, or try to figure out the transformation to apply to normalize the card art, and feed that to the CNN.