Probably Dance

I used to be an AI programmer working on video games, and I’m currently trying to learn machine learning. As part of this I find myself having to repeatedly explain why video games don’t use machine learning. People seem to find it interesting enough because it’s not just the obvious reasons (machine learning is hard and far from solved for game playing) but it’s also about developer control and about making an understandable game for the player. Video game AI is designed to deliver a certain experience, which is more difficult to do with machine learning. So this blog post lists the main reasons why video game AI does not use machine learning.

Explaining the Terms

Before listing the reasons I need to explain the terms though. “Machine Learning” is a part of AI (artificial intelligence) that deals with problems that are too hard to program by hand. For example facial recognition. It’s easy for us to tell two people apart, but you’d be hard pressed to write exact instructions to do that. It might be more obvious why it’s a difficult problem if you try to do it for a different species, like for cows. For farmers it’s easy to tell their cows apart, but good luck explaining the steps to somebody who never works with cows. In machine learning, instead of trying to explain how to do things step by step, we try to come up with algorithms that teach the computer to learn things by itself. Which is much more similar to how you’d do this for humans. You’d just show them the cows and tell them which is which, and if you do that often enough, they’ll learn.

The current hotness in machine learning is “deep learning” which is just a fancy word for “big neural networks” where “neural network” is a data structure that’s somewhat inspired by the human brain. This sounds scary, but current neural networks are quite rigid. They’re similar in rigidity to the neurons connecting your eyes to your brain. There’s not much thinking going on there, just translation of information.

Currently it feels like machine learning is slowly taking over the field of AI. For example chess AI used to not be machine learning. Instead you’d try to come up with clever algorithms that try to determine the best move for a given chess position. But the current best chess AI is a mix of a traditional algorithm and machine learning, where the “machine learning” part is used to learn which moves might be important and which moves can probably be ignored. The classical algorithm then trusts the machine learning part and only looks at the “important” moves when trying to determine which move is best.

Anyway with the introductions out of the way, here are the main reasons why video game AI does not use machine learning:

1. Machine Learning is Hard and Far From Solved for Game Playing

This is the biggest, most obvious reason. Progress is being made, like a recent AI that got really good at StarCraft 2, but that took years for some of the best researchers in the world. Maybe in the near future this will be easy enough that your average game studio can do this, but up to now it was not possible.

2. Game AI has to be Easy to Debug and Change

Let’s say we solved the first problem, you would immediately run into a problem of having a hard time changing your AI. If a combat designer comes to you and says “I saw the AI do X and I would really like it to not do that” then what do you do? You need to be able to find out why it did X and change that logic. In machine learning that can be very difficult. Let’s say “X” is that the AI chased a player for too long. So maybe you could try to detect that, add a negative reward for when the AI chases the player for too long, and then re-train your neural network. This may or may not work. Because maybe the AI hasn’t learned any alternative strategies to fall back to, so since this is still the best option, it’ll still do it, no matter how negative the reward. Or maybe it will no longer chase at all. Because it’s not like the AI has a neat “chase” state that you can identify, so maybe your negative reward discourages too much behavior. Or maybe you already added other positive rewards that drown out the new negative rewards that you just added.

In classical AI your enemy behavior is probably in a state machine that has a “chase” state and you can just keep a counter of how long the AI has been in that state. If that counter gets too big, you can cancel the chase.

Another example is that when DeepMind first showed the StarCraft 2 AI that I mentioned above, it was only able to play one faction on one map. Current neural networks are so rigid that if you had tried to make that AI play on a different map, or with a different faction, you wouldn’t have gotten an average AI, you would have gotten an AI that plays complete nonsense. They could have retrained that AI to play on different maps or with different races, but it requires new training every time, which takes days or weeks. Of course while you’re still developing the game, if a designer comes to you with a new map, they want to be able to play on it right away. (later versions of the same AI seem to be more flexible, but neural networks don’t make that easy)

3. Game AI Shouldn’t be too Easy, too Difficult or too Weird

In the example from the previous point, why would a designer come to you and say that the AI shouldn’t chase for too long? Because that makes the game more fun. In most games the AI will leave you alone if you run away for long enough. But no machine learning algorithm would ever learn that as a behavior. It doesn’t make sense to stop looking for you just because they haven’t seen you for 30 seconds. It only makes sense in game rules.

Can you train a machine learning algorithm to not be too difficult? Yes, but it’s actually harder to do that than to just make it really good. You have to mess with the incentives, and in my experience when you’re trying to mess with the incentives the AI swings between three different states: too difficult, too easy, and behaving weird. It can be hard to tweak the AI to land in a middle ground. This is not unique to machine learning. My experience on this actually comes from classical AI algorithms that just get complex. (like trying to write a poker AI or a chess AI that’s not too difficult and not too easy and doesn’t play weird) But those algorithms can be understood because you wrote them, so you at least have a chance to make targeted changes and to debug weirdness. With machine learning you’re one level further removed.

4. Game AI has to be Understandable

The points above are that as a developer you have to understand your AI. But actually this is also true for the player. If you’re playing a game and the AI does something weird once, you can maybe ignore that. But if it does something weird too often, players stop taking the AI seriously. They stop really interacting with it. This killed the game Really Bad Chess for me. Usually the AI is fine in that game, but sometimes it makes weirdly bad moves, like not taking a piece that you were willing to sacrifice. (example: you have two queens, the opponent has one queen. You’re willing to sacrifice one of your queens to capture the opponent’s last queen. Then the AI decides to not capture your “sacrifice”, so you just safely retreat your queen. Leaving you with two queens, the opponent with none)

As you climb up the rankings, at some point you reach a point where you and the AI should be roughly evenly matched. But the weird behavior gets in the way. Sometimes you advance when you shouldn’t because the AI behaves weird. Then the game gets too difficult and you have to lose a game to be ranked down again to the “right” ranking. If this happens too often, you’re mostly playing matches that are too difficult or artificially easy because the AI behaved weird. At some point you develop a feeling for what the AI can or can’t handle so you start taking advantage of bad behavior, but now you’re playing a different game. You’re playing the “exploit weird AI quirks” game, which is a less fun game to play. If you start playing normally again you have to lose a few games to be ranked “correctly” again, but the fun is gone, and the temptation to exploit the AI is always there when you’re presented with a tough situation.

So good games often have AI with clear rules that’s understandable. This can lead to weird behavior like in this video:

Even though the AI behaves badly in this video, it’s actually very good overall. Not “very good” as in “very intelligent,” but “very good” as in “makes for a game that’s hard at first, but possible to become very skilled at.” Both the developers and the player are aware of the rules for the AI. So the game is designed in a way that you can never get a big advantage by exploiting the AI like this. Instead the developers often built interesting puzzles where you can perfect your behavior. The predictability is important to allow you to get very good at the game, and that allowed the developers to make the game harder, more rewarding.

5. Developers Want to Design Challenges

That last point is even more explicit in the Dark Souls board game, where the opponents are just controlled by the rules in the manual. Their behavior is entirely predictable. As an AI programmer your job is similar to the people who designed the rules for the Dark Souls board game: You have to design interesting challenges, not actually make smart enemies.

I talked above about an AI that got really good at StarCraft 2. That AI would be terrible for the single player game. In most single player StarCraft 2 levels you’re presented with overwhelming enemies that you slowly take apart. If those enemies were controlled by AI that was actually smart, they would defeat you very quickly. Instead you want an AI that sends an attack wave at regular intervals, slowly increasing in difficulty so that the player can prepare for them.

Could you design a game that would be fun to play against the strong StarCraft 2 AI? Probably, but you might be more limited in the levels that you can build.

Part of this is thinking about what the point of games is. It’s not to bang your head against the AI, it’s to practice life skills. The AI is just one of the tools in the game’s toolbox to help you do that.

Where Machine Learning is Used in Video Games

All that being said, machine learning has been used in video games and is slowly being used more, just not for enemy AI.

Car Driving

I think machine learning is used to find the ideal route on race tracks in racing games. It’s a nice constrained optimization problem that machine learning can solve. And it might be hard to solve programmatically since it’s slightly different for every car.

We often had trouble just tweaking all the parameters for cars in our games, just to get the AI to drive normally. Whenever a new car came in, the other AI programmer on the team had to spend some time making the AI work well with the new car. Why not just do the same thing for every car? Because you’d have to cheat, and if you don’t cheat, the AI makes more realistic mistakes. “Cheating” when driving a car means that given “the AI wants to drive at X speed in this direction” you’d just completely ignore the car physics and just force the car to go at that speed in that direction. But if you want the AI to make mistakes like “it approached that curve too quickly and had to brake hard at the last moment, slowing it down too much” you have to manually program that into your cheating model. And suddenly you have to emulate car physics in your cheating model and you end up with something really complicated. Where if you just make the AI use the same controls that the player uses, you can get these natural looking mistakes for free. But it does means that you have to do a bit of work for different cars that control differently. The AI programmer who worked on this was convinced that machine learning could be used to tweak these parameters, but we never got around to trying that.

Animation Clean Up

Most animation these days is done using motion capture. The problem is that motion capture data is always a bit noisy. So animators have to “clean up” the data when it first comes in. That is repetitive work and is now being taken over by machine learning.

Default Animation and Voices

Machine learning recently got good at generating voices and animations. That’s now being used to fill in default voices and facial animations for games. As a writer on a game you often get very few iterations. Maybe a line can be re-recorded once or twice, but that’s the limit. So instead you use robot voices before getting a real voice actor in the studio. But robot voices are just too bad, not giving you a real experience of what your lines sound like. An alternate approach is to just have people in the studio record the lines, but that also takes time. Now robot voices are getting good enough where you can just use them as your temp audio, maybe even good enough that you might get away with shipping robot voices for minor side characters.

You have a similar problem with conversations in RPGs like Skyrim: You have to animate the face that you’re talking to. Machine learning is also being used to get better default results for that, requiring animators to do less manual work.

This is actually important for the kinds of games you can make. The best story-heavy games tend to not be voice-acted. One reason is that voice acting is expensive, but another reason is that writers can easily change text, but not easily change voices. And you may want to rewrite the text after you hear it spoken out loud. All this is changing. Even if you just use the machine learning for temp voices, you can still get more fully voiced iterations in. Then, once you’re happy with the dialogue, you can record with voice actors at the end. Voice acting is still expensive, but at least you only have to do it once, (per language) and at least you were able to arrive at high quality dialogue through iteration before having to commit to the voice acting.

Anti Aliasing and Ray Tracing

One of the reasons why ray tracing is now possible to do in real time is that it now uses machine learning. You don’t actually use it in the actual ray math, but instead you use it to extrapolate from few rays to many rays. Instead of shooting thousands of rays per pixel, you shoot a few and then use machine learning to smooth things out.

Machine learning was also used to learn how anti aliasing works. Now we have better anti aliasing.

Future Uses of Machine Learning in Games

You’ll notice that all of these are nicely constrained problems where we can either learn from lots of examples, or we can express the problem as an optimization problem. That’s what machine learning is currently good at. And, weirdly, it’s actually not being used much for the AI in the game.

If the research into game playing continues at the current pace, we’ll have machine learning algorithms that can learn to play most games in a couple years. When that happens, and reason 1 above goes away, we may see machine learning used more in video game AI. While the other reasons won’t go away by themselves, more developers will still try every time that one of the reasons is eliminated.

At that point we’ll make different kinds of games. You can’t use an AI that was trained on machine learning for designed experiences like Sekiro or in single player StarCraft levels. So the kinds of levels we build will change. This does also mean that there will always be games that don’t use machine learning AI, simply because it wouldn’t make those kinds of games any more fun. You’ll always want games with the simple rule based AI just because that’s something you can master in a reasonable amount of time.