2017­5­18
Artificial intelligence: Why AI researchers like video games | The Economist
Artificial intelligence
Why AI researchers like video games
Games help them understand reality
Print edition | Science and technology
May 11th 2017
LAST year Artur Filipowicz, a computer scientist at Princeton University, had a
stop-sign problem. Dr Filipowicz is teaching cars how to see and interpret the
world, with a view to them being able to drive themselves around unaided. One
quality they will need is an ability to recognise stop signs. To that end, he was
trying to train an appropriate algorithm. Such training meant showing this
algorithm (or, rather, the computer running it) lots of pictures of lots of stop signs
in lots of different circumstances: old signs and new signs; clean signs and dirty
signs; signs partly obscured by lorries or buildings; signs in sunny places, in rainy
places and in foggy ones; signs in the day, at dusk and at night.
http://www.economist.com/news/science­and­technology/21721890­games­help­them­understand­reality­why­ai­researchers­video­games
1/6
2017­5­18
Artificial intelligence: Why AI researchers like video games | The Economist
Obtaining all these images from photo libraries would have been hard. Going out
into the world and shooting them in person would have been tedious. Instead, Dr
Filipowicz turned to “Grand Theft Auto V”, the most recent release of a well-known
series of video games. “Grand Theft Auto V” is controversial because of its realistic
portrayal of crime and violence—but from Dr Filipowicz’s point of view it was ideal,
because it also features realistic stop signs. By tinkering with the game’s software,
he persuaded it to spit out thousands of pictures of these signs, in all sorts of
situations, for his algorithm to digest.
Dr Filipowicz’s stop signs are one instance of the
fondness that students of artificial intelligence (AI, of
From Russia with risk
which machine vision is an example) have for video
BUTTONWOOD’S NOTEBOOK
games. There are several reasons for this popularity.
The end of Canada’s housing boom?
AMERICAS
Some people, such as Dr Filipowicz, use games as
training grounds for the real world. Others, observing
Our general election poll tracker
GRAPHIC DETAIL
that different games require different cognitive skills,
think games can help them understand how the
See all updates
problem of intelligence may be broken down into
smaller, more manageable chunks. Others still, building on these two observations,
think games can help them develop a proper theory of artificial (and perhaps even
natural) intelligence.
Latest updates
Learner driver
For all of this to happen, though, the games themselves have first to be tweaked so
that they can be played directly by another computer program, rather than by a
human being watching the action on a screen. “Grand Theft Auto V”, for instance,
can be turned from a source of pictures of road signs into a driving simulator for
autonomous vehicles by bolting onto it a piece of software called “Deep Drive”. This
lets the driving and navigation programs of such vehicles take control—a cheaper
and safer way of testing driving software than letting it loose on roads.
Games companies are beginning to understand this. In June 2015, for instance,
Microsoft started Project Malmo, an AI-development platform based on a popular
“world-building” game called “Minecraft” that it had recently purchased. In
November 2016 Activision Blizzard, owners of “Starcraft II”, a science-fiction
strategy game in which players build and command human and alien armies,
http://www.economist.com/news/science­and­technology/21721890­games­help­them­understand­reality­why­ai­researchers­video­games
2/6
2017­5­18
Artificial intelligence: Why AI researchers like video games | The Economist
announced something similar in collaboration with DeepMind, an AI firm owned
by Alphabet, Google’s holding company.
The following month, with the permission of the owners involved, a privately
financed research group in San Francisco, called OpenAI, released “Universe”. This
is a piece of software, free for all to use, which features hundreds of games
presented in ways that mean they can be played directly by appropriate programs.
The offerings in “Universe” range from bestselling, big-budget titles such as “Portal
2” (a physics-based puzzle game) to cheap-and-cheerful web games like “Bubble Hit
Pony Parade” and “James the Space Zebra”.
One of Microsoft’s hopes in starting Project Malmo was to teach AI software to
collaborate with people. To this end, Katja Hofman, the project’s head, is trying to
use “Minecraft” to create an advanced personal assistant. Her goal is software that
can anticipate what its human operator wants, and help him achieve it. “Minecraft”,
which is simpler than the real world but still complicated enough to be interesting,
makes the perfect testing-ground. Dr Hofman and her colleagues are, for instance,
using it to try to teach a computer to work out that it must co-operate with a human
player in order to catch a virtual pig. Since the machine is incapable of
understanding written instructions, it must learn co-operation purely by watching
the actions of its human confrères in the game.
Acting as training wheels for the real world is not,
however, the only thing video games can do for AI.
The fact that different games require different
talents helps researchers chop up the problem of
intelligence. In 2015 DeepMind released a paper
describing how its researchers had trained an
artificial neural network—a program based loosely on the structure of a biological
brain—to play dozens of different games released in the 1970s and 1980s by Atari, a
pioneering video-games company.
Some games proved harder than others for the network to master. “Breakout”,
which is a bit like a single-player version of tennis, was easy. The objective is to
smash floating blocks by hitting them with a bouncing ball. A player can do one of
two things: move the “racket” left or move it right. Failure is punished instantly
(missing the ball costs a life). Similarly, success is instantly rewarded (each
smashed block adds to the score). This combination of simplicity and immediate
http://www.economist.com/news/science­and­technology/21721890­games­help­them­understand­reality­why­ai­researchers­video­games
3/6
2017­5­18
Artificial intelligence: Why AI researchers like video games | The Economist
feedback suited DeepMind’s neural network, which learnt to play “Breakout” so
well that it reached scores more than ten times those a professional human gamestester can manage.
Other games were less straightforward. In “Montezuma’s Revenge” the goal is to
retrieve treasure buried deep inside a danger-filled pyramid. To do this players
must first achieve lots of sub-goals, such as finding keys to open doors. Feedback is
less immediate than in “Breakout”—for instance, a key that turns up in one area
might open a door in another, far away. And the ultimate reward, reaching the
treasure, is the consequence of thousands of previous actions. This meant that the
network found it hard to connect cause and effect. In contrast to its virtuoso
performance at “Breakout”, it was able to make almost no headway at all with
“Montezuma’s Revenge”.
Since then, DeepMind’s researchers have tweaked their algorithms to make the
system more curious about things, by giving it bigger rewards for exploration and
experimentation. This makes it more likely to stumble across good strategies which
have payouts that are not immediately apparent. That approach is not limit to
mastering skills in a virtual world—it can be applied to the real one, as well.
DeepMind’s algorithms have, for instance, been put to use in Google’s data centres,
where they have developed ways to cut energy use by 40%. Indeed, it is possible to
view tasks like that as games in themselves. To cut energy use in a data centre, a
network can tweak things like coolant-pump settings and load distributions while
keeping an eye on energy use. The lower it can get the “score”, the better it is doing.
Embodiments of truth
At the moment, repurposing a games-playing program to run a data centre’s energy
budget really is like teaching it a new game from scratch. That is because
DeepMind’s original neural network could learn to play only one game at a time. In
order to understand “Breakout”, for example, it would have to forget everything it
knew about “Space Invaders”. Such amnesia is in the nature of artificial neural
networks—and is something that distinguishes them from real brains. They learn
by system-wide adjustments of the strengths of the connections between the
virtual neurons of which they are composed. Change the task to be learned, and the
old web of connections will gradually be overwritten. Now, however, as they
describe in a paper published in March, DeepMind’s programmers have worked out
how to overcome this and let a network master many games at once, in the way that
http://www.economist.com/news/science­and­technology/21721890­games­help­them­understand­reality­why­ai­researchers­video­games
4/6
2017­5­18
Artificial intelligence: Why AI researchers like video games | The Economist
a real brain can. That is a step towards transfer learning—the ability to put to use in
one context patterns of behaviour learned in another—which is a hot topic in AI
research.
Like displaying curiosity and delaying rewards, transferring learning from one task
to another is something humans do effortlessly but machines struggle to manage.
Here again, games are playing an important role in research. For example, Julian
Togelius of New York University has organised a challenge called the General Video
Game AI Competition. Entrants must create a single program that can play, with
reasonable competence, ten different video games that neither it nor its
programmers have come across. This requires the software to master many skills—
planning, exploration, decision-making and so on—and apply them to problems it
has not previously encountered.
Even when transfer learning is mastered, though, constructing useful artificial
intelligence will remain a piecemeal activity. What researchers would really like is
an underlying theory of how to do so systematically. One candidate to be such a
theory, called embodied cognition, argues that, instead of trying to design
intelligence into a program from the beginning, it needs to be learned entirely from
experience.
Dr Hofman, in particular, backs this approach. She reckons video games are perfect
platforms on which to explore the idea. Previous attempts to study embodied
cognition, carried out in the 1980s, involved fitting robots with sensors and letting
them learn, by running around and bumping into things, how the real world works.
Researchers back then did have some success with this approach, but they ran into
problems scaling their experiments up. As David Silver, who works at DeepMind,
observes: “Robots have gears and wheels and motors, and all sorts of fiddly things
like that. You end up spending a lot of time doing maintenance work.”
Play up, play up and play the game
Video games can streamline this process. A virtual robot in a virtual world is
weightless. It has no moving parts, so needs no maintenance. Adjusting it to
change its specifications does not require breaking out the spanners and taking it
to bits. A few strokes on a keyboard will suffice.
Its environment can be altered easily, too. Rerouting a maze no longer means
welding sheets of metal together or gluing plastic walls. And a computer can run
http://www.economist.com/news/science­and­technology/21721890­games­help­them­understand­reality­why­ai­researchers­video­games
5/6
2017­5­18
Artificial intelligence: Why AI researchers like video games | The Economist
thousands of such simulations at a time, allowing legions of virtual robots to try
tasks again and again, learning with each attempt. That kind of large-scale testing,
which permits the learning process itself to be monitored and understood, is
simply not practical using real machines.
The important thing, according to Demis Hassabis, DeepMind’s founder, is to make
sure the virtual robot cannot cheat. It must navigate using only the information its
virtual sensors can gather. There can be no peeking behind the scenes of the
simulation. If such a robot wants to learn its way around the danger-filled pyramid
in “Montezuma’s Revenge”, or the fictional city of Los Santos in “Grand Theft Auto”,
it must work out where it is and what is happening from what it can “see”, not by
asking the computer which is running the game to give it co-ordinates. This is the
approach DeepMind takes when it teaches programs to play video games.
Studying embodied cognition in this way is a logical conclusion of the gamesplaying approach to AI. It seems an appropriate one. Watch the young of any
intelligent creature, from dogs to humans, and you will see them building up
something that looks suspiciously like embodied cognition by playing. Evolution
did not have the assistance of computers when it arrived at this process. But the
fundamental point of such activity, in both the artificial and the natural worlds, is
to prepare players for the biggest game of all—reality.
This article appeared in the Science and technology section of the print edition under the headline "Shall we play
a game?"
http://www.economist.com/news/science­and­technology/21721890­games­help­them­understand­reality­why­ai­researchers­video­games
6/6