Verification, Validation, and
Accreditation (VV&A) of Social
Simulations
Deborah Duong
Augustine Consulting
TRAC Monterey
Is VV&A of Social Simulations any Different
from VV&A of Physics-Based Simulations?
• Differences in the state of the art and in the
subject matter drive differences in VV&A
• Biggest difference: greater uncertainty
• Greater epistemic uncertainty
• Social scientists don’t agree on social environment
• Lack of consensus on computer representation of social
phenomena
• Lack of controlled experiments
• Greater intrinsic uncertainty
• Humans are “the symbolic species,” and symbols are
arbitrary
• Greater path dependence in social (complex adaptive)
systems
Computer Science Experimentation as a tool to
Develop Social Theory
• Theory tells what is arbitrary and what is
patterned
• Accurate VV&A will help to develop better social
theories
• New technologies can tease out cause without real
world experiments, and despite intrinsic
uncertainty
• Intrinsic uncertainty can never be solved, but if
we have an accurate measure of it, we can make
informed bets
• VV&A needs numerical measures of confidence
A New Iterative Process for Social Science
• Construct a formalized theory called a Conceptual
Model
• Generality and parsimony
• Implement the theory in computer simulation
• Data Farm feasible regions of the input space
• Draw correspondence from response surface to
real world data the model hasn’t seen before
• Obtain numerical measure of fit to data
• Repeat, with goal of maximizing fit
Gaps in ability to do Social Science with the
Computer
• Separation of assumptions from output
• Separation of testing set from training set
• Ability to focus on feasible regions of input
space in data farming
• Careful design of conceptual model
• Ability to draw correspondence to real world
data
• Ability to obtain measure of fit to lots of
partially matching data
Separation of Assumptions from Output
• In order to be valid for the analysis, the assumptions can not
lead directly to the output
• Descriptive vs. Generative Models
• Descriptive models can not be used for analysis
• For example, a book which leads you to different sections depending
on the outcome
• “If you choose to pick up the gold coin, go to page 50. If you leave it lay, go
to page 75”
• Good for Training Simulations because they tell consequences of
choices.
• Generative Models
• Not directed, computed from first principles
• For example, a primordial soup which starts with few known
assumptions, from which results are computed
• Most social models fall somewhere in between
New Technology to Bridge the Gap: Cognitive
Agent Based Simulation
• Agent Based simulations must walk through assumptions
• Not as flexible as System Dynamics or other descriptive simulations
that offer little resistance to incorrect theories
• One goal of Social Simulation in DoD is to test how international
interventions affect institutions
• We want interventions change the natural system in place so that when
we pull out, our improvements remain in place naturally
• Cognitive agents model emergence of institutions based on motivation
• Agents may learn different strategies to seek goals in reaction to
interventions
• Higher order effects emerge when they also they react to other agents
reactions
• Vicious and Virtuous cycles of emergent institutions in agent based
simulations are not directed: if they were, they would not be legitimate
analysis
Separation of the Testing Set from the Training
Set
• If we want to use a simulation in a situation it has
never seen before, we should have a measure of
confidence in how its done with other situations
it hasn’t seen before (Before it is used to inform
real world decisions!)
• This is the same as testing the ability to generalize
• Theory tells us what is general and what is not
• Parsimonious theories are correct theories, even in
the social sciences
• The Ptolemaic universe is not general or parsimonious,
The Copernican is general and parsimonious
Ability to focus on feasible regions of input
space in data farming
• Computer design of experiments can create a
simulation response surface to compare to real world
data
• Finding feasible input regions is an important focus of
data farming research
• Issue: in complex dynamic systems, like social systems,
cause is path dependent and arises during the simulation
• One solution: Strategic Data Farming, focusing response
surface on feasible areas as the simulation unfolds.
• Social systems are coevolutionary and have moving fitness
landscapes.
Careful design of Conceptual Model
• Scientific theories need a formal representation for the
scientific community to refute
• In science by simulation, that representation is a conceptual
model
• The conceptual model defines what parts of the simulation
are open to refutation, that are supposed to have fidelity
with the world, and what parts are implementations
• In a composed system, many models can be used to
implement a single conceptual model for a study
• Different input sets into a model may be accredited for
different studies, as determined by the conceptual model for
the individual study
• Conceptual models are functional, telling “what” not “how”
Conceptual Model Precision
• Rule of thumb: if it matters to the output, it should be in the
conceptual model
• Precise conceptual models help the simulation to be artifact-proof
• Model “Docking” helps us to know what implementations make a
difference to the output
• For example, continuous vs. discrete time steps
• Imprecise conceptual models need more verification
• Passing verification, that a social model has been correctly
implemented, is never trivial!
• Conceptual Models are Functional, and Implementations depends on the
existence of technologies to tell “how”
• We do not have the technology to pass the Turing Test
• Determining that an imprecise model is correctly implemented is even
harder
Example of Hypothesis Refutation based on
Conceptual Model using Schelling Model
• Hypothesis: Prejudice is not necessary for
segregation of housing to occur
• Challenge: Your result depends on geometries of
houses that don’t have fidelity in the world
• Case 1: The geometry is in the conceptual model.
Claim is refuted in validation phase, because
something that is supposed to have fidelity to the
world does not.
• Case 2: The geometry is not in the conceptual model.
Claim is refuted in the verification phase, because the
geometry makes a difference to the result, and thus
should be in the conceptual model.
Ability to draw correspondence to real world
data
• Data Data Everywhere but not a drop to drink!
• Social models can use almost any data on the
internet
• No data is exactly corresponding, but lots of
data is a close match
• Data mediation is needed between simulations
and data that can draw a correspondence
including closeness of match under uncertainty
New Technology to Bridge Gap: Ontologies
• Ontologies make a good formalization of the conceptual
model
• Ontologies categorize data in taxonomies from general to
specific, and tell the relations between categories
• In a composed systems, ontologies ensure semantic
interoperation and enforce the conceptual model contract,
accrediting only certain inputs to each model for the study.
Ontologies “wrap” the models.
• Study conceptual model and model conceptual model
wrappers can form hub and spoke mediations
• Study conceptual model and real-world data ontologies can
also be mediated in hub and spoke fashion, to draw a
correspondence
Ability to obtain measure of fit to lots of
partially matching data
• If lots of corresponding data were made available
for comparison, even though it matched only
partially, it could be used to calculate a numerical
match to simulation study data
• Technologies of soft computation and optimization can
solve this problem
• If we already trusted one set of simulations, we
could use output from these models to find a
measure of fit to other simulations we are testing
• This match is a measure of confidence
Summary
• We can not predict outcomes, but as in medicine,
we can categorize types of social states in need of
treatment, types of cures, and gather statistics and
confidence levels on how often they work
• Uncertainty will always exist, but knowledge of
confidence levels can help us make informed
decisions
• New technologies can assist validation by making
data available and measuring fit to data