Michael Ummels received his diploma degree in computer science from RWTH Aachen
University. He started his doctoral studies at the same university in 2006, supervised
by Prof. Dr. Erich Grädel and Prof. Dr. Dr.h.c. Wolfgang Thomas. As of February 2010,
the author is a postdoctoral researcher at ENS Cachan.
Stochastic Multiplayer Games: Theory and Algorithms
Stochastic games provide a versatile model for reactive systems that are
affected by random events. This dissertation advances the algorithmic theory
of stochastic games to incorporate multiple players, whose objectives are not
necessarily conflicting. The basis of this work is a comprehensive complexitytheoretic analysis of the standard game-theoretic solution concepts in the
context of stochastic games over a finite state space. One main result is that
the constrained existence of a Nash equilibrium becomes undecidable in this
setting. This impossibility result is accompanied by several positive results,
including efficient algorithms for natural special cases.
Stochastic Multiplayer Games
Theory and Algorithms
Michael Ummels
Michael Ummels
Stochastic Multiplayer Games
Theory and Algorithms
Typeset by the author in Fedra Serif B using LATEX.
Cover design by Sam Ross-Gower.
ISBN 978 90 8555 040 2
NUR 918
D 82 (Diss. RWTH Aachen University, 2010)
© Michael Ummels, 2010.
Published by Pallas Publications, Amsterdam University Press, Amsterdam.
cbnd
This work is licensed under the Creative Commons Attribution-NonCommercialNoDerivs 3.0 Unported License.
To view a copy of this license, visit http://
creativecommons.org/licenses/by-nc-nd/3.0/, or send a letter to Creative Commons, 171 2nd Street, Suite 300, San Francisco, California, 94105, USA.
Stochastic Multiplayer Games:
Theory and Algorithms
Von der Fakultät für Mathematik, Informatik und
Naturwissenschaften der RWTH Aachen University
zur Erlangung des akademischen Grades eines Doktors
der Naturwissenschaften genehmigte Dissertation
vorgelegt von
Diplom-Informatiker
Michael Ummels
aus Köln
Berichter: Universitätsprofessor Dr. Erich Grädel
Universitätsprofessor Dr. Wolfgang Thomas
Assistant Professor Dr. Marcin Jurdziński
Tag der mündlichen Prüfung: 27. Januar 2010
Diese Dissertation ist auf den Internetseiten der Hochschulbibliothek online verfügbar.
Preface
The last decades have seen an immense amount of research on the algorithmic content of game theory. On the one hand, a new subject called
algorithmic game theory has emerged that is concerned with the study of the
algorithmic theory of finite games with multiple players. On the other hand,
infinite and, in particular, stochastic two-player zero-sum games have become
an important tool for the verification of open systems, which interact with
their environment.
The aim of this work is to bring together algorithmic game theory with
the games that are used in verification by extending the algorithmic theory of stochastic two-player zero-sum games to incorporate multiple players, whose objectives are not necessarily conflicting. In particular, this
work contains a comprehensive study of the complexity of the most prominent solution concepts that are applicable in this setting, namely Nash and
subgame-perfect equilibria.
This book is the result of my doctoral studies at RWTH Aachen University. I am indebted to my primary supervisor Erich Grädel for giving me the
opportunity to pursue these studies, for introducing me to the scientific community and for giving me advice just when I needed it. I am equally grateful
to my secondary supervisor Wolfgang Thomas for his constant support and
encouragement.
Marcin Jurdziński did not hesitate to act as an external reviewer for this
thesis. I thank him not only for his careful reading and numerous remarks,
but also for giving an inspiring talk on branching vector addition systems,
which indirectly led to the resolution of a problem that was left open in the
original version of this thesis.
A substantial part of this book is based on joint work with Dominik Wojtczak. I am indebted to him for our numerous illuminating discussions,
for his insights and ideas, and—last but not least—for hosting me in Edinburgh, Amsterdam and Oxford.
5
Among the various other people who contributed to this work, I would
like to thank in particular Łukasz Kaiser for many enlightening discussions
and for discovering Proposition 3.18. Special thanks also go to Florian Horn
for many interesting discussions, to János Flesch for pointing out Proposition 3.13, and to Peter Bro Miltersen for drawing my attention to Corollary 4.4.
Moreover, I am grateful to Hugo Gimbert and Eilon Solan for answering my
questions and to Rohit Chadha, Tobias Ganzow, Jörg Olschewski and Edeline
Wong for their comments on preliminary drafts of this work.
Finally, I would like to thank Sam Ross-Gower for designing the cover of
this book, and Donald Knuth and Leslie Lamport for creating (LA)TEX.
Paris, November 2010
6
Contents
1 Introduction • 15
1.1 Games and equilibria • 15
1.2 The stochastic dining philosophers problem • 21
1.3 Contributions • 25
1.4 Related work • 27
1.5 Outline • 28
2 Stochastic Games • 31
2.1 Arenas and objectives • 31
2.2 Strategies and strategy profiles • 37
2.3 Subarenas and end components • 41
2.4 Values, determinacy and optimal strategies • 42
2.5 Algorithmic problems • 47
2.6 Existence of residually optimal strategies • 51
3 Equilibria • 55
3.1 Definitions and basic properties • 55
3.2 Existence of Nash equilibria • 59
3.3 Existence of subgame-perfect equilibria • 64
3.4 Computing equilibria • 69
3.5 Decision problems • 73
4 Complexity of Equilibria • 77
4.1 Positional equilibria • 77
4.2 Stationary equilibria • 82
4.3 Pure and randomised equilibria • 88
4.4 Finite-state equilibria • 96
4.5 Summary of results • 98
7
Contents
5 Decidable Fragments • 99
5.1 The strictly qualitative fragment • 99
5.2 The positive-one fragment • 113
5.3 The qualitative fragment for deterministic games • 122
5.4 Summary of results • 133
6 Conclusion • 135
6.1 Summary and open problems • 135
6.2 Perspectives • 138
A Preliminaries • 141
A.1 Probability theory • 141
A.2 Computational complexity • 144
B Markov Chains and Markov Decision Processes • 149
B.1 Markov chains • 149
B.2 Markov decision processes • 152
Bibliography • 157
Notation • 169
Index • 171
8
List of Figures
1.1 Matching pennies as a game in extensive form • 18
1.2 Dining philosophers • 22
1.3 Processes for the stochastic dining philosophers problem • 23
1.4 The stochastic dining philosophers game with two
philosophers • 24
2.1 A hierarchy of prefix-independent objectives • 36
2.2 An example of a two-player SSMG • 38
2.3 An MDP with no optimal strategy • 44
2.4 Another MDP with no optimal strategy • 44
3.1 A two-player reachability game with an irrational Nash
equilibrium • 58
3.2 A two-player game with a pair of optimal strategies that cannot
be extended to a Nash equilibrium • 62
3.3 An SSMG with no stationary Nash equilibrium • 63
3.4 A two-player SSMG with no positional Nash equilibrium • 64
3.5 A Büchi SMG with no subgame-perfect equilibrium • 68
3.6 An SSMG that has a stationary subgame-perfect equilibrium
where player 0 wins almost surely but no pure Nash equilibrium
where player 0 wins with positive probability • 75
3.7 The different decision problems related to Nash and
subgame-perfect equilibria • 76
4.1 Reducing SAT to PosNE, StatNE, PosSPE and StatSPE • 79
4.2 Reducing SqrtSum to StatNE and StatSPE • 85
4.3 Simulating a two-counter machine • 90
4.4 Reducing from the halting problem • 97
9