Using BotArena for Playtesting

[JarmenKell]

I have recently gotten back in to MtG, thanks to DotP. With the new DotP 2014 just around the corner, I started to wonder how I could better tune my decks this year.

I decided it would be best if I could conduct multiple AI vs AI matches and collect data about the matches to determine what cards worked better than others. Since DotP decks have limited variability in their construction, you could compare various deck builds and collect information quickly.

After browsing the forums a bit, I decided BotArena was the only program I could use that would simulate multiple AI matches and produce log files that I could parse to collect information. So far this is working well, I have started a Google Site to detail my work and provide the results of the simulations. There is already some preliminary data up as well but everything is in its infancy. At the moment I am just looking at the 4 basic deck builds in DotP 2014 that have been leaked: Avacyn's Glory, Enter the Dracomancer, Chant of Mul Daya, and Masks of Dimir. I have patched the few missing cards with lands.

So far the information gleaned looks okay. I am conducting 100 head to head battles of the different decks and parsing the log files using Python. I have seen the AI perform some odd plays and the Avacyn deck can frequently crash the simulation, especially when exiling cards, but I have noticed that setting the "Maximum thinking time" to "Test Mode" and unchecking "Re-think in Combat Phase" prevents this, though I am not sure what those settings mean. Despite, I still see some questionale plays as well as cards disappearing when they are tapped in the UI.

What I would like from the community is some discussion on this. Does this seem like a good way to playtest decks? Is there some data that would be helpful in determining if a card is critical to a deck? What would be some good builds to test after the first basic builds? What am I doing wrong?

The spam filters are blocking the website url, simply google "Magic the Shattering". It should be the first one.

[FranAvalon]

I am glad that you find BA usefull to do that kind of test. Maybe you can set BA's thinking time to 15 sec and check if it crashes.

On the other hand, I've tried to check your website and I couldn't.

[JarmenKell]

Not sure what the problem with the website is. Since I still can't post links in my posts I am not sure how to share my data on this forum or verify that you are at the correct website.

15 seconds still results in a crash on the Avacyn's Glory Deck, it seems to be a problem with exiling cards. Test Mode is the only way to get the simulations to work.

[FranAvalon]

Are you taking about Fiend Hunter and Oblivion ring?
I was testing a version of the Avacyn's deck, setting AI to 30 sec and I didn't get game crashes.

This is the deck that I've used

1 Angelic Overseer
1 Archangel
1 Champion of the Parish
2 Doomed Traveler
2 Elder Cathar
2 Angelic Overseer
2 Elgaud Inquisitor
2 Fiend Hunter
1 Geist-Honored Monk
2 Goldnight Commander
1 Mentor of the Meek
1 Midnight Guard
1 Requiem Angel
2 Selfless Cathar
3 Thraben Valiant
2 Unruly Mob
1 Voice of the Provinces
3 Gather the Townsfolk
1 Honor of the Pure
2 Oblivion Ring
4 Path to Exile
25 Plains


I've changed Hallow (not supported yet) for a couple of Path to Exile.
As you can see on the attached gamelog, the game starts with a Fiend Hunter and Oblivion ring war. (the log is from an AI vs AI mirror game)

[JarmenKell]

Correct, the build I used was similar but the crash only occurs when I run multiple simulations in a row. It will typically crash after 4-6 games but never when I run just one. I usually try to run 10 to a 100 games at once.

[FranAvalon]

I see, thx for the test.
Maybe its a memory leak or something similar. Sadly I don't have enough skill to trace it.

[JarmenKell]

I feel that is most likely the cause of the crash. I can try to recreate the problem and post the logs but I don't think it will help track down the source of the error.

Here're the links to both the main page and the simulation data.

Main Page:
https://sites.google.com/site/magictheshattering/home

Data:
https://sites.google.com/site/magictheshattering/documents