It is currently 23 Oct 2017, 19:03
   
Text Size

How can I run AI tournaments with a set of AIs/decks?

Moderators: melvin, beholder, ubeefx, ShawnieBoy, Lodici, CCGHQ Admins

How can I run AI tournaments with a set of AIs/decks?

Postby shedletsky » 11 Sep 2016, 21:19

I've seen Project Firemind, which is interesting, but very slow throughput. Is there a way to setup my own firemind workers to handle my own firemind duels (and other people can use them when I'm not?). This combined with some sort of API to batch-upload firemind tournament requests would be ideal.

Alternatively, I have read some threads that suggest Magarena can be configured to allow for AI vs AI matches. I haven't seen an option for this in the GUI, so I assume this has to do with writing a batch script to queue up all the matches I want, and then parsing the results into something readable. How would I go about setting this up?

I want to genetically tune random decks and have them fight against each other in millions of matches. I don't mind writing code to glue various parts together, but I would love to know where the parts are.
shedletsky
 
Posts: 7
Joined: 11 Sep 2016, 19:56
Has thanked: 0 time
Been thanked: 0 time

Re: How can I run AI tournaments with a set of AIs/decks?

Postby melvin » 12 Sep 2016, 02:02

shedletsky wrote:I've seen Project Firemind, which is interesting, but very slow throughput. Is there a way to setup my own firemind workers to handle my own firemind duels (and other people can use them when I'm not?). This combined with some sort of API to batch-upload firemind tournament requests would be ideal.
That's a neat idea. Since firemind knows your access key, this should be possible. @mike may have more info on this.

shedletsky wrote:Alternatively, I have read some threads that suggest Magarena can be configured to allow for AI vs AI matches. I haven't seen an option for this in the GUI, so I assume this has to do with writing a batch script to queue up all the matches I want, and then parsing the results into something readable. How would I go about setting this up?
Yes, this is possible and that is how we did the AI comparison. It is command line only though.
Code: Select all
cd <Magarena directory>
java -cp Magarena.jar magic.DeckStrCal \
  --deck1 <path to .dec file> \
  --ai1 [MMAB|MMABC|MCTS|MCTSC] \
  --str1 [1-8] \
  --deck2 <path to .dec file> \
  --ai2 [MMAB|MMABC|MCTS|MCTSC] \
  --str2 [1-8]
MMAB - minimax
MCTS - monte carlo tree search
MMABC, MCTSC - version of AI that knows all hidden information

shedletsky wrote:I want to genetically tune random decks and have them fight against each other in millions of matches. I don't mind writing code to glue various parts together, but I would love to know where the parts are.
Sounds like a fun project, all the best. Do post your progress here on the forum.
User avatar
melvin
AI Programmer
 
Posts: 1011
Joined: 21 Mar 2010, 12:26
Location: Singapore
Has thanked: 36 times
Been thanked: 437 times

Re: How can I run AI tournaments with a set of AIs/decks?

Postby shedletsky » 12 Sep 2016, 07:32

Thanks melvin, that is really helpful.

Are there docs for setting up a pool of firemind workers outside of the magarena client?

I have a crapton of Azure credits that I plan to burn through.
shedletsky
 
Posts: 7
Joined: 11 Sep 2016, 19:56
Has thanked: 0 time
Been thanked: 0 time

Re: How can I run AI tournaments with a set of AIs/decks?

Postby melvin » 12 Sep 2016, 10:05

shedletsky wrote:Are there docs for setting up a pool of firemind workers outside of the magarena client?
As far as I know, the firemind client/server communication is not published. You should send mike (developer of Firemind) a private message about this or ask on https://github.com/firemind/ProjectFire ... ues/issues

shedletsky wrote:I have a crapton of Azure credits that I plan to burn through.
For reference, previously I used https://en.wikipedia.org/wiki/Oracle_Grid_Engine to submit command line scripts to a compute cluster. Perhaps something similar can be setup on Azure.
User avatar
melvin
AI Programmer
 
Posts: 1011
Joined: 21 Mar 2010, 12:26
Location: Singapore
Has thanked: 36 times
Been thanked: 437 times

Re: How can I run AI tournaments with a set of AIs/decks?

Postby mike » 21 Sep 2016, 15:27

I know I'm a little late to the party but I thought I'd answer your questions anyway to have this documented.

Is there a way to setup my own firemind workers to handle my own firemind duels (and other people can use them when I'm not?).
There is an option to run the worker through the Magarena UI via Settings -> Run Firemind Worker

You can get the access key on firemind.ch on your profile page.

If you want to automate running the worker on the command line you can set the key in release/Magarena/general.cfg as firemindAccessToken=XYZ

You can start the queue worker with this command

Code: Select all
java -noverify -cp Magarena.jar magic.FiremindQueueWorker
This combined with some sort of API to batch-upload firemind tournament requests would be ideal.
There is currently no API call to create duels. I opened a ticket for this.


I have a crapton of Azure credits that I plan to burn through.
I'm using a VM running ubuntu for my workers. Azure offers ubuntu servers so it should be easy enough to set up. Let me know if you need setup specifics.
User avatar
mike
Programmer
 
Posts: 126
Joined: 05 Jul 2013, 17:00
Has thanked: 0 time
Been thanked: 28 times


Return to Magarena

Who is online

Users browsing this forum: No registered users and 3 guests


Who is online

In total there are 3 users online :: 0 registered, 0 hidden and 3 guests (based on users active over the past 10 minutes)
Most users ever online was 279 on 11 Jul 2013, 22:03

Users browsing this forum: No registered users and 3 guests

Login Form