World Deck Development thread
Moderators: timmermac, Blacksmith, KrazyTheFox, Agetian, friarsol, CCGHQ Admins
Re: World Deck Development thread
by chops » 22 May 2014, 19:56
100 decks is a lot, but I can run a test with one game per match-up to see how long that takes.
Oh, my program doesn't like decks with ":" in their name, and they're all over these decks. I can code something to replace them with "-". I knew that was a problem I'd run into sooner or later. (I use ":" to determine where the scores are in Forge's log-files.)
Oh, my program doesn't like decks with ":" in their name, and they're all over these decks. I can code something to replace them with "-". I knew that was a problem I'd run into sooner or later. (I use ":" to determine where the scores are in Forge's log-files.)
Re: World Deck Development thread
by Xyx » 22 May 2014, 22:05
I noticed a lot of variance while testing myself, though, and the one winning the die roll is generally quite advantaged as well, so if you could do more games, that would really improve the quality of the results.chops wrote:I can run a test with one game per match-up to see how long that takes.
Works for me!chops wrote:my program doesn't like decks with ":" in their name, and they're all over these decks. I can code something to replace them with "-".
Thanks for giving it a shot! I was considering running an AI tournament, but that would probably have taken me several days.
Re: World Deck Development thread
by chops » 23 May 2014, 07:35
- PT Journey to Nyx decks | Open
Margin of error varies from deck to deck, but is generally between 0.9 and 1.0. Each deck got played 99 times though, so the margin of error is similar.
I'm glad I didn't do more than this many to start with. Even playing one game of each of the 4,900 match-ups took several hours. I can run it a couple more times while I'm at work tomorrow if you want.
Re: World Deck Development thread
by Xyx » 23 May 2014, 08:08
Awesome! Thanks! 
99 games should be more than enough. I hadn't realized you'd make them all play against each other. That's a lot of games! I assumed you'd use Swiss pairings, but this is better. It's like 99 rounds of Swiss.
It seems that the Forge AI does not agree with the outcome of the Pro Tour. The ChannelFireball BUG Control decks that put up such good results are crowding the bottom of the list whereas the top is mostly aggro; the exact opposite of the actual results.
How should I interpret the results? The top score of 29.5, would that suggest a "Hard" difficulty? Or would that be "Very Hard" since it's close to the best Forge can do? And would the bottom score of 20.3 be "Medium" or "Hard"?
Time for me to prune all the lower-scoring versions of each archetype. These will make for some excellent opponents!
99 games should be more than enough. I hadn't realized you'd make them all play against each other. That's a lot of games! I assumed you'd use Swiss pairings, but this is better. It's like 99 rounds of Swiss.
It seems that the Forge AI does not agree with the outcome of the Pro Tour. The ChannelFireball BUG Control decks that put up such good results are crowding the bottom of the list whereas the top is mostly aggro; the exact opposite of the actual results.
How should I interpret the results? The top score of 29.5, would that suggest a "Hard" difficulty? Or would that be "Very Hard" since it's close to the best Forge can do? And would the bottom score of 20.3 be "Medium" or "Hard"?
Time for me to prune all the lower-scoring versions of each archetype. These will make for some excellent opponents!
Re: World Deck Development thread
by chops » 23 May 2014, 09:14
Swiss is efficient for deciding a tournament winner, but round-robin is more accurate for determining which decks are best against the entire meta. I might experiment with Swiss later.
Interesting that the tournament results are so different. I suppose we already knew that the Forge AI doesn't do so well at control decks. Shuhei Nakamura and Paul Rietzl's BUG Control decks seem significantly better for the AI, so there's some hope there. UW Control by Luis Gutierrez even made the top 20.
It's up to you how to interpret this exactly. If you're not going to let the Very Hard decks "cheat" by using non-block cards, then the top few decks are probably close to the best you'll get. I could play these 100 decks against a few decks from the default quest to see how they compare. That's a bit more coding, but it's a feature I intended to do soon anyway.
After playing with the data a bit, I believe you can be most confident in the ordering of the decks at the top and bottom of the list. Towards the middle, the order is less reliable.
Also, the clustering of scores in the 20-30 range is probably due to the randomness of MtG. If the game were Chess, I would expect them to fill nearly the whole range of 0-50.
Interesting that the tournament results are so different. I suppose we already knew that the Forge AI doesn't do so well at control decks. Shuhei Nakamura and Paul Rietzl's BUG Control decks seem significantly better for the AI, so there's some hope there. UW Control by Luis Gutierrez even made the top 20.
It's up to you how to interpret this exactly. If you're not going to let the Very Hard decks "cheat" by using non-block cards, then the top few decks are probably close to the best you'll get. I could play these 100 decks against a few decks from the default quest to see how they compare. That's a bit more coding, but it's a feature I intended to do soon anyway.
After playing with the data a bit, I believe you can be most confident in the ordering of the decks at the top and bottom of the list. Towards the middle, the order is less reliable.
Also, the clustering of scores in the 20-30 range is probably due to the randomness of MtG. If the game were Chess, I would expect them to fill nearly the whole range of 0-50.
Re: World Deck Development thread
by serrasmurf » 23 May 2014, 10:46
Hi Chops,
Thanks a lot! Awesome!
Control/ combo decks score even worse than I thought. Ai versus AI is imo very representative for human versus AI.
There are 2 factors that define the opposition: deck & playing. Comparing your data against the protour data (or other big samples) nicely isolates the AI factor.
An interesting excercise would be to experiment with both factors (deck variations & additional deck specific instructions for the AI how to play (not sure what the possibilites are, i'm not a developer)) and then run your program again, to optimize AI & deck.
KR
Thanks a lot! Awesome!
Control/ combo decks score even worse than I thought. Ai versus AI is imo very representative for human versus AI.
There are 2 factors that define the opposition: deck & playing. Comparing your data against the protour data (or other big samples) nicely isolates the AI factor.
An interesting excercise would be to experiment with both factors (deck variations & additional deck specific instructions for the AI how to play (not sure what the possibilites are, i'm not a developer)) and then run your program again, to optimize AI & deck.
KR
- serrasmurf
- Posts: 316
- Joined: 30 Jan 2010, 14:09
- Location: The Netherlands
- Has thanked: 3 times
- Been thanked: 18 times
Re: World Deck Development thread
by chops » 23 May 2014, 21:04
Absolutely! If a dev wanted to tweak the AI, I'd be happy to post my scripts. The only reason I haven't done that yet is that they'll only work on my machine right now, because they look for things in a folder with my name in it. I'm sure that's fixable though.serrasmurf wrote:An interesting excercise would be to experiment with both factors (deck variations & additional deck specific instructions for the AI how to play (not sure what the possibilites are, i'm not a developer)) and then run your program again, to optimize AI & deck.
KR
Re: World Deck Development thread
by friarsol » 24 May 2014, 16:38
chops, I'd be interested in looking at your code. If you can change the parameters so you can pass in locations instead of being static we could probably put in the tools directory where we put other things like the SetTracking scripts that we use around release time.
- friarsol
- Global Moderator
- Posts: 7593
- Joined: 15 May 2010, 04:20
- Has thanked: 243 times
- Been thanked: 965 times
Re: World Deck Development thread
by chops » 24 May 2014, 21:32
I'll take a look at doing that tonight. This is all a learning experience for me, but I don't mind learning a few more things.
Would you prefer them to print results to the command line, or to a log file? I've got a bit of both so far, and it's annoying even me.
Would you prefer them to print results to the command line, or to a log file? I've got a bit of both so far, and it's annoying even me.
Re: World Deck Development thread
by friarsol » 24 May 2014, 22:52
Maybe major things to command line (starting new matches, final match record/score) so there's a general feeling of progress as it runs, and all the specific data to file (potentially multiple files for ease of differentiation), that sound manageable?chops wrote:I'll take a look at doing that tonight. This is all a learning experience for me, but I don't mind learning a few more things.
Would you prefer them to print results to the command line, or to a log file? I've got a bit of both so far, and it's annoying even me.
- friarsol
- Global Moderator
- Posts: 7593
- Joined: 15 May 2010, 04:20
- Has thanked: 243 times
- Been thanked: 965 times
Re: World Deck Development thread
by chops » 25 May 2014, 07:02
Makes sense.
Oh, one other thing is that I use a Windows-specific command to check if Forge is still running. I could look into solutions for other OS's, but the most efficient solution would be to tweak Forge. Right now Forge's CLI returns immediately. If Forge didn't return until the match was over, that would eliminate my scripts' busy-waiting. Is that an easy fix on your end, or should I stick with what I have?
Oh, one other thing is that I use a Windows-specific command to check if Forge is still running. I could look into solutions for other OS's, but the most efficient solution would be to tweak Forge. Right now Forge's CLI returns immediately. If Forge didn't return until the match was over, that would eliminate my scripts' busy-waiting. Is that an easy fix on your end, or should I stick with what I have?
Re: World Deck Development thread
by friarsol » 31 May 2014, 02:19
Well, I haven't really looked at how the CLI works in the codebase, but I'd say for now not too worry about that and just post what you have (since it sounds functional) and then if someone is able to make the appropriate changes you can make your script more OS-agnostic.chops wrote:Makes sense.
Oh, one other thing is that I use a Windows-specific command to check if Forge is still running. I could look into solutions for other OS's, but the most efficient solution would be to tweak Forge. Right now Forge's CLI returns immediately. If Forge didn't return until the match was over, that would eliminate my scripts' busy-waiting. Is that an easy fix on your end, or should I stick with what I have?
- friarsol
- Global Moderator
- Posts: 7593
- Joined: 15 May 2010, 04:20
- Has thanked: 243 times
- Been thanked: 965 times
Re: World Deck Development thread
by serrasmurf » 04 Jun 2014, 11:31
Will Theros World be part of the next release?
- serrasmurf
- Posts: 316
- Joined: 30 Jan 2010, 14:09
- Location: The Netherlands
- Has thanked: 3 times
- Been thanked: 18 times
Re: World Deck Development thread
by Xyx » 05 Jun 2014, 14:06
If you're referring to the Theros world I'm working on, then... maybe? I'm more or less done making sure the decks work, but there are still some AI related issues that I don't really want to solve by cutting those cards from all of the decks entirely. I would like to maintain a "100% card coverage". I'm not sure if that is acceptable.
I still need to sort out some of the Block Constructed decks. GP Manchester was Block Constructed and a few new and interesting deck types popped up there. After that, all that remains is fine-tuning the difficulty levels.
I haven't made any challenge decks. The ideal challenge decks would be the official ones (Face the Hydra, Battle the Horde, Defeat a God), but those aren't currently supported. I could whip up some placeholder stuff if necessary.
I'll sort out those GP Manchester decks and upload what I have somewhere in the next 20 hours, and then I'll leave it up to the powers that be. If it's not a problem to update the world in the next release, then I suggest including it. I don't expect any significant changes on my part anymore.
I still need to sort out some of the Block Constructed decks. GP Manchester was Block Constructed and a few new and interesting deck types popped up there. After that, all that remains is fine-tuning the difficulty levels.
I haven't made any challenge decks. The ideal challenge decks would be the official ones (Face the Hydra, Battle the Horde, Defeat a God), but those aren't currently supported. I could whip up some placeholder stuff if necessary.
I'll sort out those GP Manchester decks and upload what I have somewhere in the next 20 hours, and then I'll leave it up to the powers that be. If it's not a problem to update the world in the next release, then I suggest including it. I don't expect any significant changes on my part anymore.
Who is online
Users browsing this forum: No registered users and 25 guests