Verification and validation of agent-based models:
Epistemological importance and theoretical challenges
Dr. Claudius Gräbner
Institute for the Comprehensive Analysis of the Economy (ICAE)
Johannes Kepler University Linz
www.claudius-graebner.com
Agent-based modeling in Ecological Economics - From toy model to verified tool of analysis
ESCP Europe Business School
19.05.2017
1
Outline
I. Why taking an epistemological perspective?
II. How can ABM create knowledge about reality?
III. What is the role of model verification and validation?
IV. Practical implications
V. Are verification & validation necessary?
VI. Summary and outlook
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
2 /24
Why taking an epistemological perspective?
Epistemology is the study of knowledge and justified belief*
What are the necessary and sufficient conditions of knowledge?
What are the sources of knowledge?
How can we generate knowledge about the real world?
Always important when choosing and justifying one’s modeling framework
Indispensable for relating results from different models
Helps identifying advantages and disadvantages of different modeling frameworks
Clarifies what validation and verification is about, why it is important, and what we
can expect from it
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
*: Steup, M. (2016): Epistemology, in: E. Zalta (ed.). The Stanford
Encyclopedia of Philosophy.
3 /24
Outline
I. Why taking an epistemological perspective?
II. How can ABM create knowledge about reality?
III. What is the role of model verification and
validation?
IV. Practical implications
V. Summary and outlook
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
4 /24
A basic epistemological framework
Further development of Mäki’s Models as Isolations and Surrogate Systems (MISS, Mäki 2009)
r: R –> R
Reality
R0
Representation
Surrogate
R1
g: R –> S
Inference of
mechanisms
h: S –> R
S0
Inference of states
S1
s: S –> S
Models necessary representations of reality that isolate aspects
Exploring the behavior of the model hopefully explains reality
Can refer to good predictions or the identification of effects
Can refer to identifying mechanisms
General movement that prioritizes mechanism-based explanations (e.g. Deaton,
2010; Grüne-Yannoff, 2016; Gräbner, 2017)
Claudius Gräbner - Theoretical assessment of ABM - 20.05.2016
Mäki, U. (2009). MISSing the World. Models as Isolations and
Credible Surrogate Systems, Erkenntnis, Vol. 70(1), pp. 29-43.
5 /17
Outline
I. Why taking an epistemological perspective?
II. How can ABM create knowledge about reality?
III. What is the role of model verification and
validation?
IV. Practical implications
V. Summary and outlook
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
6 /24
The role of verification and validation
r: R –> R
Reality
R0
Representation
Surrogate
g: R –> S
R1
Inference of
mechanisms
h: S –> R
S0
Inference of facts
S1
s: S –> S
Verification: Testing whether the model does what it should do.
Validation: Testing whether the model resembles the SUI.
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
7 /24
Verification
Surrogate
S0
S1
s: S –> S
As is often the case, confirming that the
model was correctly programmed was
substantially more work than
programming the model in the first
place.
Removed picture because of
copyright, see:
http://wwwpersonal.umich.edu/~axe/
Robert Axelrod (1997)
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
8 /24
Verification
Surrogate
S0
S1
s: S –> S
What makes verification easy?
A simple and transparent model structure
The simpler the model, the easier verification
What methods do we have for verification?
Analytical proofs
Anti-bugging, degeneracy-testing, structured walk-through
In particular: DOE
Sloppy verification can severely damage a whole research program
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
9 /24
Validation I
r: R –> R
Reality
R0
Representation
Surrogate
R1
g: R –> S
Inference of
mechanisms
S0
h: S –> R
Inference of facts
S1
There are different ways of validating a model (Tesfatsion, 2017)
1. Input validation
2. Process validation
3. Descriptive output validation
4. Predictive output validation
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
Tesfatsion, L. (2017). Modeling Economic Systems as LocallyConstructive Sequential Games" (2017). Economics Working Papers. 23.
10 /24
Validation II
r: R –> R
Reality
R0
R1
Inference of
mechanisms
Surrogate
1.
S0
h: S –> R
Inference of facts
S1
Input validation
Are the exogenous inputs meaningful?
Requires an explicit representation of
real-world entities in the model
Direct representations better than „as if
twists“
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
Serious input validation
benefits from a
sufficiently complex
model and good data.
11 /24
Validation III
r: R –> R
Reality
R0
Representation
Surrogate
2.
R1
g: R –> S
Inference of
mechanisms
S0
h: S –> R
Inference of facts
S1
Process validation
Does the model resemble real-world mechanisms?
Real mechanisms are unobservable
Function -> Mechanism is one-to-many
Can only be conjectured (Bunge, 2004)
PV cannot happen on quantitative grounds alone
Not an issue for all economists, but increasingly
acknowledged for many reasons (e.g. Deaton, 2010)
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
Serious process
validation requires an
explicit representation
of real-world entities in
the model and benefits
from modular design.
12 /24
Validation IV
r: R –> R
Reality
R0
Representation
Surrogate
3.
R1
g: R –> S
Inference of
mechanisms
S0
S1
Descriptive output validation
How well can the model be calibrated to the data?
Danger: Overfitting & empirical risk minimization
Problem: Equifinality
Ceteris paribus: the
more complex, the
better!
The most prominent form of validation in economics
Why?
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
13 /24
Why that focus on descr. output validation?
Yes, it is also easier…
But after about five years of doing likelihood
ratio tests on rational expectations models, I
recall Bob Lucas and Ed Prescott both
telling me that those tests were rejecting
too many good models.
Thomas Sargent (2005)
This habit seems to have some deeper science-sociological reasons…
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
Source picture: https://de.wikipedia.org/wiki/Datei:Nobel_Prize_2011Press_Conference_KVA-DSC_7716.jpg
14 /24
Validation V
r: R –> R
Reality
R0
Representation
Surrogate
4.
R1
g: R –> S
Inference of
mechanisms
S0
S1
Predictive output validation
How well can the model be trained on the data?
Model must actually produce the output
Still not that common in economics but many
meth. innovations from machine learning
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
Requires a lot of
data, technical
knowledge, and
willingness to do it
15 /24
The relation of verification & validation
Verification
Input
validation
Process
validation
Descriptive output
validation
Predictive
output
validation
Models that are easy to verify are usually more difficult to validate
Generally, ABM seem to have…
a comparative advantage when it comes to input and process validation
a comparative disadvantage when it comes to verification
Modeling frameworks have different advantages - can be used to our advantage
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
16 /24
What kind of validation is best?
The „best“ validation technique depends on the model purpose
Aim
Input
Process
Verification
validation validation
Descript.
validation
Predictive
validation
Provide
predictions
Explain
what has
happened
Scenario
analysis
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
Epstein, J. (2008). Why model?. JASSS, 11(412)
17 /24
Outline
I. Why taking an epistemological perspective?
II. How can ABM create knowledge about reality?
III. What is the role of model verification and
validation?
IV. Practical implications
V. Summary and outlook
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
18 /24
Two practical take-aways
I. Some design aspects are never harmful and
always useful
Modularity
Transparency and replicability
II.Trade-off between verification and validation
Removed picture because of
copyright, see:
https://
image.slidesharecdn.com/
introductiontotradeoffs-13100300
2746-phpapp02/95/introductionto-tradeoffs-7-638.jpg?
cb=1444957614
Concrete specification depends on the purpose of the
model
Attractive proceeding: sequential modeling (Gräbner
et al. 2017)
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
Source of the picture: Own cat.
19 /24
Sequential model building
Combine the respective advantages of simple and complex models (Gräbner et al. 2017)
Start with a simple model, preferably analytically tractable
Acemoglu & Robinson (2002) on inequality dynamics
Proof some (marginally interesting) results
Some comparative statics results on when inequality in- and decreases
Build ABM of this model and verify it against analytical model
Increase complexity of the model sequentially
What are the inequality dynamics?
What is the role of institutions?
Respective advantages may be combined
Does work only if a drastic complexity reduction makes sense
Docking problem (Axtell et al. 1996)
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
20 /24
Is verification and validation necessary?
Verification is always important
Gerard Debreu & validation:
„A […] theory has a mathematical form that is completely
separated from its economic content. […] the divorce of form
and content.“
Accordingly, for the model builder, validation
as such is not essential
However, there are no economic models that
are never validated in some sense
Without any relation to the real world, no
model can exist
Gerard Debreu (1988)
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
Source picture: https://en.wikipedia.org/wiki/G%C3%A9rard_Debreu#/
media/File:Debreu,_G%C3%A9rard_(1921-2004).jpeg
21 /24
Is verification and validation necessary?
Nevertheless, there are many models that…
are not meant to be validated explicitly
cannot be validated explicitly
These models are still important
Building intuition
Building blocks for other models
Theory building & making observations
possible:
„It is absolutely wrong to build a theory only
on observations. Because it is only the prior
theory that decides what we can actually
observe.“
Albert Einstein in Heisenberg (1927)
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
Source of the picture: https://de.wikipedia.org/wiki/
Datei:Einstein_Albert_Elsa_LOC_32096u.jpg
22 /24
Conclusions
Verification & validation measure model properties that are essential for epistemic
meaningfulness
Verification is always important
The need of validation depends on modeling purpose
Trade-offs between verification and various forms of validation
The comparative disadvantage of ABM in economics is with verification
The comparative advantage of ABM in economics is with input and process
validation
Sometimes the respective advantages can be combined (sequential modeling)
Better validation is good, perfect validation is impossible, and thus a plurality of
models and perspectives is important
Claudius Gräbner - Epistemological perspective on verification & validation of ABM - 19.05.2017
23 /24
Thank you very much for your attention!
Acemoglu, D. & Robinson, J.A., (2002). The Political Economy of the Kuznets Curve. Review of Development Economics, 6(2), pp.1–21. https://dx.doi.org/
Axelrod, R. (1997) Advancing the Art of Simulation in the Social Sciences. In Conte R, Hegselmann R, and Terna P (Eds.) Simulating Social Phenomena, Lecture Notes in
Economics and Mathematical Systems 456: 21-40. Berlin: Springer-Verlag.
Axtell, R.L. et al., 1996. Aligning Simulation Studies: A Case Study. Computational and Mathematical Organization Theory, 1(2), p.123–141.
Bunge, M. (2004). How does it work? The search for explanatory mechanisms. Philosophy of the Social Sciences, 34(2), 182–210. https://dx.doi.org/
10.1177/0048393103262550
Deaton, A. (2010). Instruments, Randomization, and Learning about Development, Journal of Economic Literature, Vol. 48, pp. 424–455. https://dx.doi.org/10.1257/jel.48.2.424
Debreu, G. (1986). Mathematical Form and Economic Content. Econometrica, 54(6), p. 1259-1270. https://dx.doi.org/10.2307/1914299
Epstein, J. (2008). Why Model?, JASSS, 11(412). http://jasss.soc.surrey.ac.uk/11/4/12.html
Gräbner, C. (2017). The Complementary Relationship Between Institutional and Complexity Economics: The Example of Deep Mechanismic Explanations. Journal of Economic
Issues, 51(2), p.392–400. https://dx.doi.org/10.1080/00213624.2017.1320915
Grüne-Yanoff (2016). Why Behavioural Policy Needs Mechanistic Evidence, Philosophy & Economics, 32(3), p. 463-483. https://dx.doi.org/10.1017/S0266267115000425
Heisenberg (1986). Philosophie und Quantentheorie. Berlin: Reclam.
Mäki, U. (2009). MISSing the World. Models as Isolations and Credible Surrogate Systems, Erkenntnis, Vol. 70(1), pp. 29-43. https://dx.doi.org/10.1007/s10670-008-9135-9
Steup, M. (2016): Epistemology, in: E. Zalta (ed.). The Stanford Encyclopedia of Philosophy. https://plato.stanford.edu/archives/fall2016/entries/epistemology/
Tesfatsion, L. (2017). Modeling Economic Systems as Locally-Constructive Sequential Games" (2017). Economics Working Papers. 23. http://lib.dr.iastate.edu/
econ_workingpapers/23
Claudius Gräbner - Theoretical assessment of ABM - 20.05.2016
24 /24