How can I run AI tournaments with a set of AIs/decks?

by **shedletsky** » 11 Sep 2016, 21:19

I've seen Project Firemind, which is interesting, but very slow throughput. Is there a way to setup my own firemind workers to handle my own firemind duels (and other people can use them when I'm not?). This combined with some sort of API to batch-upload firemind tournament requests would be ideal.

Alternatively, I have read some threads that suggest Magarena can be configured to allow for AI vs AI matches. I haven't seen an option for this in the GUI, so I assume this has to do with writing a batch script to queue up all the matches I want, and then parsing the results into something readable. How would I go about setting this up?

I want to genetically tune random decks and have them fight against each other in millions of matches. I don't mind writing code to glue various parts together, but I would love to know where the parts are.

by **melvin** » 12 Sep 2016, 02:02

shedletsky wrote:I've seen Project Firemind, which is interesting, but very slow throughput. Is there a way to setup my own firemind workers to handle my own firemind duels (and other people can use them when I'm not?). This combined with some sort of API to batch-upload firemind tournament requests would be ideal.

That's a neat idea. Since firemind knows your access key, this should be possible. @mike may have more info on this.

shedletsky wrote:Alternatively, I have read some threads that suggest Magarena can be configured to allow for AI vs AI matches. I haven't seen an option for this in the GUI, so I assume this has to do with writing a batch script to queue up all the matches I want, and then parsing the results into something readable. How would I go about setting this up?

Yes, this is possible and that is how we did the AI comparison. It is command line only though.

Code: Select all: cd <Magarena directory> java -cp Magarena.jar magic.DeckStrCal \ --deck1 <path to .dec file> \ --ai1 [MMAB|MMABC|MCTS|MCTSC] \ --str1 [1-8] \ --deck2 <path to .dec file> \ --ai2 [MMAB|MMABC|MCTS|MCTSC] \ --str2 [1-8]

MMAB - minimax
MCTS - monte carlo tree search
MMABC, MCTSC - version of AI that knows all hidden information

shedletsky wrote:I want to genetically tune random decks and have them fight against each other in millions of matches. I don't mind writing code to glue various parts together, but I would love to know where the parts are.

Sounds like a fun project, all the best. Do post your progress here on the forum.

by **shedletsky** » 12 Sep 2016, 07:32

Thanks melvin, that is really helpful.

Are there docs for setting up a pool of firemind workers outside of the magarena client?

I have a crapton of Azure credits that I plan to burn through.

by **melvin** » 12 Sep 2016, 10:05

shedletsky wrote:Are there docs for setting up a pool of firemind workers outside of the magarena client?

As far as I know, the firemind client/server communication is not published. You should send mike (developer of Firemind) a private message about this or ask on https://github.com/firemind/ProjectFire ... ues/issues

shedletsky wrote:I have a crapton of Azure credits that I plan to burn through.

For reference, previously I used https://en.wikipedia.org/wiki/Oracle_Grid_Engine to submit command line scripts to a compute cluster. Perhaps something similar can be setup on Azure.

by **mike** » 21 Sep 2016, 15:27

I know I'm a little late to the party but I thought I'd answer your questions anyway to have this documented.

Is there a way to setup my own firemind workers to handle my own firemind duels (and other people can use them when I'm not?).

There is an option to run the worker through the Magarena UI via Settings -> Run Firemind Worker

You can get the access key on firemind.ch on your profile page.

If you want to automate running the worker on the command line you can set the key in release/Magarena/general.cfg as firemindAccessToken=XYZ

You can start the queue worker with this command

Code: Select all: java -noverify -cp Magarena.jar magic.FiremindQueueWorker

This combined with some sort of API to batch-upload firemind tournament requests would be ideal.

There is currently no API call to create duels. I opened a ticket for this.

I have a crapton of Azure credits that I plan to burn through.

I'm using a VM running ubuntu for my workers. Azure offers ubuntu servers so it should be easy enough to set up. Let me know if you need setup specifics.