Introducing ForgeScribe - the AI for scripting forge cards

[austinio7116]

I've been working on a project to save time scripting new sets when they come out. It started with the idea of using the Scryfall API to create a template card script with everything but the scripted abilities generated for each non-reprint card in a set. I then realized that we could use deep learning to try to build a neural network that could "translate" the oracle text provided on Scryfall into Forge's card scripting language and so generate a fully functional card script. Clearly this would have limitations - as every card is unique and new abilities and keywords are added in each set, but at least if it could handle some of the simpler cards it would save time on scripting a new set and allow people to focus on the new and more complex cards.

[austinio7116]

I now have a fully functional tool that can do this - we are calling it ForgeScribe. We will be testing this on the C18 spoilers when they come out to get an idea of how many cards it can accurately script. It is likely that there will be some cards that it can script completely, some that it makes minor errors in that will need correcting, some it gets very wrong and will need a complete rewrite and of course some that contain new keywords and abilities that will need code change anyway.

[austinio7116]

For those that are interested, the neural network that I developed for this uses the https://github.com/tensorflow/nmt Neural Machine Translation tensorflow code - similar to the Google Translate system. This is a sequence to sequence network that learns to convert one sequence of strings to another and is designed for automatic language translation. I had to convert the oracle text into tokenized strings as input, train the network on the 18,000 cards that are already scripted in Forge, then convert the output tokenized strings back into syntactically correct forge scripts.

[austinio7116]

Here are a few examples of the code correctly scripting some cards whose oracle text I have made up (or taken from RoboRosewater) - to ensure we are testing examples that did not exist in the training data:

Code: Select all

echo "Enchant creature\nEnchanted creature gets +1/+1 and has \"{T}: Target creature gains first strike until end of turn.\"" | /NMT/scripts/forgeScribe4.0.sh

Code: Select all

K:Enchant creature
A:SP$ Attach | Cost$ 1 R | ValidTgts$ Creature | AILogic$ Pump
S:Mode$ Continuous | Affected$ Creature.EnchantedBy | AddPower$ 1 | AddToughness$ 1 | AddAbility$ DeviantGleeTrample | Description$ Enchanted creature gets +1/+1 and has "{T}: Target creature gains first strike until end of turn."
SVar:DeviantGleeTrample:AB$ Pump | Cost$ T | ValidTgts$ Creature | TgtPrompt$ Select target creature | KW$ First Strike | SpellDescription$ Target creature gains first strike until end of turn.

[austinio7116]

Code: Select all

echo "Target Merfolk gets +9/+7 and gains hexproof and trample until end of turn. Target creature can't block this turn." | /NMT/scripts/forgeScribe4.0.sh

Code: Select all

A:SP$ Pump | Cost$ 2 R | ValidTgts$ Merfolk | TgtPrompt$ Select target Merfolk | NumAtt$ +9 | NumDef$ +7 | KW$ Hexproof & Trample | SubAbility$ DBPump2 | SpellDescription$ Target Merfolk gets +9/+7 and gains hexproof and trample until end of turn. Target creature can't block this turn.
SVar:DBPump2:DB$Pump | ValidTgts$ Creature | TgtPrompt$ Select target creature | KW$ HIDDEN CARDNAME can't block. | IsCurse$ True

[austinio7116]

First 3 C18 spoilers have been scripted by ForgeScribe - looks like a decent attempt from the AI:

Code: Select all

Name:Coveted Jewel
ManaCost:6
Types:Artifact
T:Mode$ ChangesZone | Origin$ Any | Destination$ Battlefield | ValidCard$ Card.Self | Execute$ TrigDraw | TriggerDescription$ When CARDNAME enters the battlefield, draw three cards.
SVar:TrigDraw:DB$Draw | Defined$ You | NumCards$ 3
A:AB$ Mana | Cost$ T | Produced$ Any | Amount$ 3 | SpellDescription$ Add three mana of any one color.
T:Mode$ AttackersDeclared | AttackingPlayer$ You | ValidCard$ Creature.OppCtrl | TriggerZones$ Battlefield | Execute$ TrigDraw | TriggerDescription$ Whenever one or more creatures an opponent controls attack you and aren't blocked, that player draws three cards and gains control of defender. Untap
Oracle:When Coveted Jewel enters the battlefield, draw three cards.\n{T}: Add three mana of any one color.\nWhenever one or more creatures an opponent controls attack you and aren't blocked, that player draws three cards and gains control of Coveted Jewel. Untap it.

=============================

Name:Geode Golem
ManaCost:5
Types:Artifact Creature Golem
PT:5/3
K:Trample
T:Mode$ DamageDone | ValidSource$ Creature.Legendary+YouCtrl | ValidTarget$ Player | CombatDamage$ True | Execute$ TrigPlay | TriggerZones$ Command | TriggerDescription$ Whenever CARDNAME deals combat damage to a player, you may cast your commander from the command zone without paying its mana cost.
SVar:TrigPlay:DB$ Play | Defined$ You | ValidCards$ Card.IsCommander+YouCtrl | WithoutManaCost$ True | Optional$ True | Amount$ 1
Oracle:Trample\nWhenever Geode Golem deals combat damage to a player, you may cast your commander from the command zone without paying its mana cost. (You still pay any additional costs.)

=============================

Name:Saheeli's Directive
ManaCost:X R R R
Types:Sorcery
K:Improvise
A:SP$ Dig | Cost$ X G G G | DigNum$ X | Reveal$ True | AnyNumber$ True | ChangeValid$ Artifact.cmcLEX | ChangeNum$ X | DestinationZone$ Battlefield | DestinationZone2$ Graveyard | References$ X | SpellDescription$ Reveal the top X cards of your library. You may put any number of artifact cards with converted mana cost X or less from among them onto the battlefield. Then put all cards revealed this way that weren't put onto the battlefield into your graveyard.
SVar:X:Count$xPaid
Oracle:Improvise (Each artifact you tap after you're done activating mana abilities pays for {1}.)\nReveal the top X cards of your library. You may put any number of artifact cards with converted mana cost X or less from among them onto the battlefield. Then put all cards revealed this way that weren't put onto the battlefield into your graveyard.

=============================


[austinio7116]

I'm particularly please that it managed to deal with this card:

Code: Select all

Name:Ancient Stone Idol
ManaCost:10
Types:Artifact Creature Golem
PT:12/12
K:Flash
S:Mode$ ReduceCost | ValidCard$ Card.Self | Type$ Spell | Amount$ X | EffectZone$ All | Description$ CARDNAME costs {1} less to cast for each attacking creature.
SVar:X:Count$Valid Creature.attacking
K:Trample
T:Mode$ ChangesZone | Origin$ Battlefield | Destination$ Graveyard | ValidCard$ Card.Self | Execute$ TrigToken | TriggerController$ TriggeredCardController | TriggerDescription$ When CARDNAME dies, create a 6/12 colorless Construct artifact creature token with trample.
SVar:TrigToken:DB$ Token | TokenAmount$ 1 | TokenName$ Construct | TokenTypes$ Artifact,Creature,Construct | TokenOwner$ You | TokenColors$ Colorless | TokenPower$ 6 | TokenToughness$ 12 | TokenKeywords$ Trample
SVar:SacMe:4
Oracle:Flash\nThis spell costs {1} less to cast for each attacking creature.\nTrample\nWhen Ancient Stone Idol dies, create a 6/12 colorless Construct artifact creature token with trample.


[austinio7116]

and this one:

Code: Select all

Name:Whiptongue Hydra
ManaCost:5 G
Types:Creature Lizard Hydra
PT:4/4
K:Reach
T:Mode$ ChangesZone | Origin$ Any | Destination$ Battlefield | ValidCard$ Card.Self | Execute$ TrigDestroy | TriggerDescription$ When CARDNAME enters the battlefield, destroy all creatures with flying. Put a +1/+1 counter on CARDNAME for each creature destroyed this way.
SVar:TrigDestroy:DB$DestroyAll | ValidCards$ Creature.withFlying | RememberDestroyed$ True | SubAbility$ DBPutCounter
SVar:DBPutCounter:DB$PutCounter | CounterType$ P1P1 | CounterNum$ X | References$ X
SVar:X:Remembered$Amount
SVar:NeedsToPlay:Creature.withFlying+YouDontCtrl
Oracle:Reach\nWhen Whiptongue Hydra enters the battlefield, destroy all creatures with flying. Put a +1/+1 counter on Whiptongue Hydra for each creature destroyed this way.

[Agetian]

Wow, very impressive! This is going to be of great assistance!

- Agetian

[austinio7116]

Thanks - it has struggled a bit with the C18 set, but I think that is just that the non-reprints are all fairly unique and complex - hopefully it will help even more with guilds.

[Agetian]

Yeah, makes sense Commander sets generally contain quite a lot of unique cards

- Agetian

[Arcanist]

This is interesting. I assume that you took inspiration from this article?

I would be keen to learn more technical details about this project e.g. the size of the network and the time/number of machines it took to train it. Also, have you had the opportunity to measure the performance of the tool (either on Commander 2018 or other cards)? The article I linked mentions that DeepMind's system performance was around 61% on MtG cards (which I think is actually pretty bad), and I would be curious to see how yours compares to that. Deepmind used XMage for their study so I would not necessarily expect the two systems to perform identically.

[silly freak]

Interesting article! I'm trying to reproduce the results myself (whenever I have a little spare time), and would be interested in the answer as well. I could imagine that forge script has a little less boilerplate compared to Java, and that this could be beneficial to the performance, but I don't know.

[Arcanist]

Interesting article! I'm trying to reproduce the results myself (whenever I have a little spare time), and would be interested in the answer as well. I could imagine that forge script has a little less boilerplate compared to Java, and that this could be beneficial to the performance, but I don't know.

The article is just one of many on the topic. I just selected the one which features the performance numbers since I was citing them. If you want to read more about this, the best reference is probably the Arxiv paper.

I am not sure what to expect in terms for performance to be honest. Sure there is less boilerplate, but that should not be a real issue for a neural network since that type of code can be copied verbatim from one card to the next. Surely the network should be able to get that right. That is why I was asking the question.

If the performance is significantly different, this could mean that there is probably an "optimal" way of representing cards. Conversely if performance on Forge is similar to that obtained on XMage, and that performance is way lower than typical translation problem, that could indicate either that the problem cannot be reduced to a translation task (meaning a more sophisticated approach is necessary) or that the neural network is too small. I don't work on machine translation so I don't know what the performance of a RNN-based state of the art system is.

I don't think the results featured in that paper are that interesting. It looks more like a case of trying to apply neural networks to yet another problem (and getting crappy results out of it in the process). More interesting experiments that have been run. Just a couple from the top of my head:

• Take card texts in two different languages, train a model, measure performance, and compare that to the results on card implementation. That would shed some light on the question I was discussing above.
• Take the English card text and corresponding Forge implementation, and train a model to produce the English text from the code. First, I think that would be fun. Second, that could potentially help spot cards which are incorrectly implemented (assuming performance is good enough of course).

While I was searching for the Arxiv paper above, I ended up finding a brand new paper called Improving Hearthstone AI by Combining MCTS and Supervised Learning Algorithms published only 2 days ago. Talk about a coincidence. I am mentioning this here because I think that will be of interest to people visiting this topic.

[austinio7116]

Interesting, I hadn't read that article actually, we had the idea independently. The main issue with performance/accuracy is that 18k examples is not really enough, especially as each card in magic is pretty much unique. I think 60% would be a reasonable result as the main aim is reducing the number of simple cards we need to manually script.