Drafting

[Rob Cashwalker]

It averages the previous rating with the new one.

[Sloth]

I also think that the system is flawed. If a card never gets picked because it's very bad and only one user first-picks it as a joke, it will likely keep the score 30 for a long time , since any other picks than last pick are unlikely.

If you then decide to implement the AI picking preferences, such a card would never change it's score again, since the AI will not pass it and the player won't pick it.

[Rob Cashwalker]

I'm hoping that's what the averaging should be eliminating.

Example:
If a 30 is averaged with a 2, then the new score will be 16. If it is later rated a 20, then the new score becomes 18. If it is later rated a 30, we get a new score of 24. If it's later rated a 1, then the new score is 12.5....

[friarsol]

I'm hoping that's what the averaging should be eliminating.

Example:
If a 30 is averaged with a 2, then the new score will be 16. If it is later rated a 20, then the new score becomes 18. If it is later rated a 30, we get a new score of 24. If it's later rated a 1, then the new score is 12.5....

Shouldn't the average be (oldRating * numberOfRatings + newRating) / (numberOfRatings+1)

[delirimouse]

I've been following this thread for a while, with great interest because I tend to do a couple drafts on Forge per day. I created an account just to post about this.

The data that has been produced seems very fishy. I think an easy (and reasonable) thing to keep track of is "What is the average slot in which a human picked this card?"

Seed this by putting in a 14 for every card, so it will always be defined. Then, each time a card is picked, it will update the average accordingly. A card like Ancestral Recall will very quickly climb, and something like Viridian Harvest will fall rapidly. Experienced drafters often describe how good a card is by talking about what slot it is usually picked.

This figure won't be completely sufficient, but it will assuredly pick out the cards that should almost never be passed.

The biggest problem with this this indicator is that if the AI is using it, it is difficult for the AI to correct its opinion of a card downwards. For example, say someone misclicked and first picked a Viridian Harvest. Now the AI thinks Viridian Harvest should be picked somewhere in the 7ish range. As a result, no human will ever pick a Viridian Harvest again because the AIs will have taken it too early.

Anyway, I'm not really a programmer, but I am a mathematician, so if you are interested in help thinking about how the drafting/deckbuilding algorithm uses the data that you collect, I would be happy to think about it and share some thoughts and outline some ideas. If you're happy with what you've got, I won't keep yapping too much! I'm really very pleased with Forge overall.

[Rob Cashwalker]

Welcome to the Forge, delirimouse.

The only trouble with basing the score on 14 is custom packs have fewer cards. the score I had worked out was based on the total cards in a draft, not per pack. The ai wouldn't necessarily always pick the highest rated card it sees ALL the time... some chaos always has to be added. I would have the ai try to pick three contenders, based on various criteria. Best creature, best artifact, best spell, best hate-pick... etc. Then randomly pick.

I'd certainly welcome alternate algorithms for rating.

[juzamjedi]

Friarsol's suggestion should improve the rating by making the ratings more stable long term. That seems like a definite change to me.

In order to evolve the AI needs to get negative feedback. Maybe the rating becomes previous rating * 0.99 or something similar so we see ratings decay over time.

[delirimouse]

The 14 could be adjusted based on the size of the packs in question.

I think you are correct that with a little randomness inserted the AI would eventually learn the true value of all the cards. There are a few cards the AI should never not take, but the AI will know this because the human will always pick them first.

Here is an algorithm that is neutral to the number of cards in a pack in a given format, and will figure out both of it undervalues a card and if it overvalues a card. We want to provide a list of the strength of cards by color (but have no desire, for the purposes of this algorithm, to compare between colors). This won't work for multicolor cards (which I sense are a very difficult problem).

When the human picks a card, assign a score of 100 to the card picked, and a score of 0 to each card of the same color not picked. If a card is the only card of its color in the pack, just assign no scores from this pick. Each card's overall score, then, should be the average of its scores. Thus, if there is no card of the same color you would pick over a given card, its score will be 100. If there is no card of a given color that would pick under a card, its score will be 0. Other cards scores will be somewhere in between.

A similar list could be maintained for colorless cards, or you could assign colorless cards a score as though they were every color (so they received a score whenever a card was picked).

I think this is easy to implement and will give you a good pick order for each color, which the AI can use to make its picks once it has decided on color (with some variance as previously suggested).

Thus, I'd suggest a decision process like this:

1. Am I already in two colors? If so, pick the best rated card in one of those two colors (including artifacts).

2. If I'm not in two colors, do I really like one of the cards in the one color I have? If so, pick that.

3. If I'm here, I pick the best card regardless of color, and now I consider myself in that color.

To avoid this inflexibility resulting in occasional serious problems, you can also force that no two adjacent AIs can be in the same color, or add a panic mode where the AI can realize it is getting really bad picks in its colors and switch.

I think it is key, though, that the AI try hard to stick to two colors. I often see the AI playing all 5 now.

I think this very simplistic notion would be sufficient to make the AI into moderately good competition in M12 draft. This is an admirable goal, since core sets should be the easiest.

[Rob Cashwalker]

Here's the scoring algorithm I'm using now:

Code: Select all

    public void setChoice(Card c) {
        CardList list = pack[getMod()];

        if (!list.contains(c))
            throw new RuntimeException("BoosterDraft : setChoice() error - card not found - " + c + " - booster pack = " + list);

        if (Constant.Runtime.UpldDrft[0]) {
            for (int i = 0; i < list.size(); i++) {
                Card cc = list.get(i);
                String CnBk = cc.getName() + "|" + cc.getCurSetCode();

                float pickValue = 0;
                if (cc.equals(c))
                    pickValue = (float)list.size() * (((((float)stopCount - (float)currentCount) * 100) / (float)stopCount) / 50);
                else
                    pickValue = 0;

                if (!draftPicks.containsKey(CnBk)) {
                    draftPicks.put(CnBk, pickValue);
                } else {
                    float curValue = draftPicks.get(CnBk);
                    float newValue = (curValue + pickValue) / 2;
                    draftPicks.put(CnBk, newValue);
                }
            }
        }

        list.remove(c);
        currentCount++;
    }//setChoice()


setChoice is called each time the human picks a card, it is passed the card the player chose. list is the pack of cards presented to the player from which the card was chosen. stopCount is the total number of cards to be drafted; currentCount is the pick we're adding right now. draftPicks is a HashMap relating the cardName + setCode to its pickValue.

About the pseudo-average.... At this level, dividing by 2 is bad because it punishes a higher picked card that you just happen to get again later in the draft.. like a good common. At the server-level, the score should be more reactive to trends. A card that was previously low, if drafted high enough for a few consecutive drafts, can bring its score up quickly, but never all the way to the top. I was hoping floating point numbers would help provide distance between card values, but I see that there are finite increments for them.

When the human picks a card, assign a score of 100 to the card picked, and a score of 0 to each card of the same color not picked. If a card is the only card of its color in the pack, just assign no scores from this pick. Each card's overall score, then, should be the average of its scores. Thus, if there is no card of the same color you would pick over a given card, its score will be 100. If there is no card of a given color that would pick under a card, its score will be 0. Other cards scores will be somewhere in between.

A similar list could be maintained for colorless cards, or you could assign colorless cards a score as though they were every color (so they received a score whenever a card was picked).

I think this is easy to implement and will give you a good pick order for each color, which the AI can use to make its picks once it has decided on color (with some variance as previously suggested).

Sounds reasonable, the AI can't really draft an archetype, but color combinations, sure.. An extension of this, would be to actually track what colors the human has been picking. The max score of 100 could be weighted based on pick timing and relevance to the primary colors picked. This would allow multi-color and colorless cards to have an impact without special treatment.

The only drawback to this method is it's a lot of data to track once it's generated. The server side SQL is going to be tricky, but manageable. On the client side, I was hoping to actually inject the values into the cardname.txt files as SVars. It would only need a few data points for full and custom formats, plus 1 for each set that the card had been drafted in. This advanced algorithm would require a local copy of the database for all possible color ratings in each format...

[delirimouse]

As far as your current algorithm, one thing you could do is instead of taking the flat average:

float newValue = (curValue + pickValue) / 2;

you could take a weighted average:

float newValue = (19*curValue + pickValue) / 20;

This makes recent picks matter more than old ones, but is not so vulnerable to sudden shocks. I just picked a weighting of 19:1, but any weighting could do. I certainly think the weighting of the current value should be at least 9 times the weighting of the new pick value.

[delirimouse]

Sounds reasonable, the AI can't really draft an archetype, but color combinations, sure.. An extension of this, would be to actually track what colors the human has been picking. The max score of 100 could be weighted based on pick timing and relevance to the primary colors picked. This would allow multi-color and colorless cards to have an impact without special treatment.

In the very long run you could teach the AI to figure out archetypes by having it observe the correlation of humans picking two cards together. For example, the AI could learn from the data fairly quickly that Cystbearer goes much better with Plague Stinger than Carapace Forger, even though Carapace Forger is the same color.

I think this would require a ton of data management to find out, though, and would likely not result in too substantial an improvement. I think drafting the best cards in a couple colors and building a deck with a reasonable number of creatures and non-creatures will probably result in a nice challenge.

One other thought, while I'm here, is that if the AI gets to the end of the draft and somehow doesn't have the cards to build a deck in two/three colors, you could just have it cheat a little and add some random decent cards in its colors to fill it out. The same could be done if it doesn't have enough creatures.

The only drawback to this method is it's a lot of data to track once it's generated. The server side SQL is going to be tricky, but manageable. On the client side, I was hoping to actually inject the values into the cardname.txt files as SVars. It would only need a few data points for full and custom formats, plus 1 for each set that the card had been drafted in. This advanced algorithm would require a local copy of the database for all possible color ratings in each format...

It isn't really a lot of data. If you look at it, all you are tracking for each card is the number of times it has been picked and the number of times it has been in a pack when a card of the same color was picked. This is just storing two integers for each card for each format, and the score (the way I outlined it) is the quotient of those two times 100 (so that it looks human readable). I think this is less storage than is going on now, since it uses two ints instead of a float.

(Edited because my formula was wrong.) To be clear, if:

X= # of times this card has been picked
Y= # of times this card has been in a pack where a card of the same color (including possibly this one) was picked

Then:

MyRank = 100 * X / Y

[lord of 13]

I faced similar issues with dynamic card ratings when creating DraftMaster. I wrote a bit about my rating system and NPC logic, and most of it is rather similar. However, I would ignore the player's ratings until 5 or 10 ratings have been accumulated, and until then give each card a rating based on age and rarity (assuming a higher rating is better, perhaps [years old] * [rarity, where Basic = 0, Common = 1, Uncommon = 2, Rare = 3 and Mythic = 5... or something].
Alternatively, if you have a deck evaluation function, run a bunch of drafts using an all-AI system to train the AI with values for each card. You'd need tens of thousands of runs, but the training data would then be pretty useful, although the cards added thereafter would merit there own hundreds of runs per new release.

[Accountancy]

Is there any that forge could check which drafted cards actually get played in the human's deck? If so, we could have this affect the score so cards like Angel's Feather will receive virtually no rating in M12 draft and Darksteel Relic in Scars.
i.e. if a card is put in the deck, then it could add 1 to it's score and if it's excluded from the deck, then you could subtract 1 from it's score, or maybe even halve it, not sure if that would affect scores too drastically though, but it would mean cards that get picked and played a lot would steadily go up, whereas cards that never get played would go down very quickily and then almost plateau, until they start going up again.

[delirimouse]

I think that if your rating system is correct this won't be necessary. If you look at my above suggestion, for example, the cards you suggest will likely have a rating at or near 0.

Furthermore, if you include whether or not a card gets played you are adding a lot of noise to the data. For example, I'll often play a Coral Merfolk because I am short on creatures. Sometimes I will first pick a Grave Titan only to see no more black cards, and then I won't play it.

How likely I am to play a card has very little to do with where I'll pick it. For example, I'll play 100% of Ponders that I draft (as long as I'm in blue) but I very rarely pick them up until very late. This is because while Ponder replaces itself and gives you a minor effect for a very low mana cost, you are only really getting a very minor effect.

What could be done is to have a separate stat on how often a card is played once it is in a pool, and have that stat used to aid in deck building.

[Accountancy]

I know this isn't on the subject of the drafting AI, but it's still drafting:
In the last two nightly builds, someone removed the ability to right-click to take a card rather than press the "choose card" button, could we get this back?