2015 48th Hawaii International Conference on System Sciences
Boundary Objects for Participatory Group Model Building of Agent-based
Models
Johnie Rose
Case Western Reserve University
[email protected]
Laura Homa
Case Western Reserve University
[email protected]
Peter Hovmand
Washington University
[email protected]
Alison Kraus
Washington University
[email protected]
Kelly Burgess
Case Western Reserve University
[email protected]
Anindita Biswas
Case Western Reserve University
[email protected]
Heide Aungst
Case Western Reserve University
[email protected]
Sarah Cherng
University of Michigan
[email protected]
Rick Riolo
University of Michigan
[email protected]
Kurt C. Stange
Case Western Reserve University
[email protected]
emerging from the actions and interactions of
individuals within a responsive environment[1, 2].
Agent-based modeling takes a mechanistic approach,
attempting to re-create the dynamics of an actual
system as opposed to simple description [2-5].
An attractive aspect of the ABM approach is the
intuitive structure that is easily understood by
stakeholders who are not experts in modeling. The
development of such models has typically been the
domain of modeling experts, however. In contrast,
another type of systems modeling approach—system
dynamics (SD)—has long been coupled with a
participatory approach to model building in which
stakeholders meet with modelers and facilitators to
iteratively turn their mental models of systems into
simulation models. SD models re-create feedbacks
that are hypothesized to generate a dynamic behavior.
In these models, emergent behavior arises from the
interaction of nonlinear balancing and reinforcing
feedback mechanisms over time [6]. SD models are
less helpful compared to ABMs for representing the
heterogeneity of individuals or the impacts of random
events.[1, 7, 8]
While participatory group model building (GMB)
of SD models has occurred for over two decades[9],
there is little documented experience with group model
building of ABMs[8]. Coupling ABM with a GMB
process could allow researchers and decision makers to
re-create systems from the level of heterogeneous
individuals through the guidance of those with intimate
system knowledge. The ultimate result would be better
Abstract
Boundary objects in participatory group processes
help participants share their understanding of systems
across disciplinary and social boundaries. We report
development of a series of boundary objects in a group
model building project in which stakeholders and
modelers constructed agent-based models (ABMs) of
health care seeking and delivery.
We propose
conventions for describing agent actions and
interactions, and offer advice for effective use of
boundary objects in a group process of building ABMs.
We found that 1) an early boundary object was helpful
for teaching stakeholders basic ABM concepts; 2)
physical correspondence between boundary objects
and
model
interfaces
improved
participant
engagement; 3) group ownership of the model was
threatened when the modeling team failed to capture
needed model elements through the boundary object;
4) adherence to boundary object conventions improved
when participants were given specific charges; 5) with
hands-on exposure, the model graphical user interface
became an effective boundary object.
1. Introduction
Agent-based models (ABMs) are a type of systems
model used increasingly in the social sciences,
biological sciences, and beyond to model the outcomes
1530-1605/15 $31.00 © 2015 IEEE
DOI 10.1109/HICSS.2015.357
2950
models, better reflecting truth, and leading to better
decisions.
Essential to this GMB process is a physical means
through which diverse stakeholders can share their
understandings of the system at hand across
disciplinary or sociocultural boundaries.
Such
boundary objects for use in GMB of SD models have
been well described and include, most prominently,
behavior-over-time charts, causal-loop diagrams, and
stock-and-flow diagrams [9]. Effective boundary
objects should be capable of portraying important
dependencies within a system, tangible with a minimal
amount of text, and modifiable and adaptable by all
group members [10, 11].
With these objects,
participants can translate what is otherwise tacit
knowledge of a system that is often inaccessible to
members of other groups into explicit knowledge that
can be universally understood and integrated by all
participants [12].
There are special challenges to developing a
boundary object for agent-based modeling. The
emphases on agent-level decisions and multi-scale
processes make the development of effective boundary
objects for ABM difficult. Whereas SD model
mapping requires depiction of a small set of actions
and effects at an aggregate level, ABM model mapping
requires depiction of possibly heterogeneous agent
decision-making rules leading to actions in a
responsive environment. The contribution of these
agent actions to an aggregate effect must then be
modeled. ABM is therefore a multi-level approach by
definition, suggesting a need for a more complex set of
boundary objects than typically used in system
dynamics.
Conventions do exist for describing the content of
agent-based models.
For instance statecharts, a
component of standard UML® (Unified Modeling
Language, uml.org), have been advocated as a means
to graphically depict the transitions of agents between
different states of being over time [13]. Perhaps better
known is a standard proposed in 2006 by Grimm et. al.
and subsequently applied in dozens of publications:
ODD (Overview, Design Concepts, and Details) [14,
15]. The goal of the ODD developers was to
standardize published ABM descriptions to help ensure
completeness, understandability, and reproducibility.
While ODD is a tremendously valuable tool to
document and describe the operationalization of
ABMs, it is not suitable as a boundary object. Instead,
its intent is to describe ABMs which have already been
developed and not necessarily as a tool for use during
model
development.
ODD-compliant
model
descriptions require a significant amount of text
targeted toward those who would program a similar
model [15]. As such, neither UML statecharts nor
ODD meet the criteria of completeness, relative
parsimony, and transformability required of effective
boundary objects.
Here, we set out to provide guidance to
investigators wishing to develop agent-based models
using a participatory GMB process. We begin by
reporting our experience with a particular GMB project
in which we held a series of eight working sessions
with patient, caregiver, and primary care provider
stakeholders to develop an ABM of health care seeking
and delivery. We describe a set of conventions for
depicting agent actions and interactions, and we
describe the series of boundary objects that evolved
throughout development of this model and discuss
their strengths and limitations. We conclude by
offering generalized recommendations for the effective
use of boundary objects in the development of ABMs
through GMB.
This work draws in part upon a theoretical
foundation of design science.
A fundamental
motivation of design science lies in enhancing the
capabilities of a group by creating and applying useful
artifacts to be applied by the group. Design science
holds that theory/truth inform design and vice
versa.[16] In the work we describe, we sought to
identify certain truths about a phenomenon-namely
health care seeking and delivery-through a design
process. We believe that the lessons we share here will
be useful to others engaged in group design processes
aimed at distilling theory or truth about a complex
system.
2. Methods
We convened a series of eight working
sessions approximately monthly for a period of nine
months. Participants were six primary care clinicians
and nine patients or family caregivers (15 participants
in total) recruited from local safety net practices in
Cleveland, OH and from ResearchMatch.org. The goal
for the group was to develop an agent-based model of
health care seeking and delivery which helped explain
the mechanisms and processes by which primary care
adds value in terms of individual and community
health beyond the treatment of illness alone.
The design of sessions is described in detail
elsewhere (pending) as are results and insights from
the model itself (pending). Briefly, the plan was to
spend the first session describing the goal as well as
introducing and demonstrating agent-based modeling.
The second session was spent developing a simple,
preliminary model based on interaction of the core
modeling team with participants. This preliminary
model was demonstrated in the third session, revised
extensively based on participant input, and presented
2951
back to the group in the following session. The same
pattern was repeated for the fourth and fifth sessions.
In the sixth session, the model was used to test various
hypotheses that had emerged over the course of
previous sessions. Based on discussion of results,
further revisions to the model were planned. In session
seven, additional hypotheses were tested using a newly
revised model, and next steps for the group—including
dissemination of findings—were discussed. In the
final session, learning was consolidated, and
participants were given practice describing the
methodology and findings in a way that would enable
them to share with others in the community.
2.1 Starting conventions
boundary object
for
an
introducing the drawing conventions shown in Table 1
to participants, the scenario below was given to
participants. As each element was described, it was
drawn on a white board using the conventions.
“We have a world. In modeling, we call it an
environment. In this environment, we have people, and
these people we call agents. In this world, there are
certain facts of the universe. One of these facts is that
each day, a person can get sick or injured. We’re
going to say that one of the facts of the world is that
there are only two ways that people need health care.
They either cut themselves and need stitches, or their
diabetes is out of control. We’ll signify cutting
yourself and needing stitches by green, and we’ll
signal out-of-control diabetes by red. There are
actions that these agents can take in response. They
can go to one type of doctor – a family doctor. One of
the facts of the world in this model is that family
doctors are better at treating diabetes. Another action
that these agents can take is that they can go to an
emergency room or ER. One of the other facts of the
world is that ER doctors are better at sewing up cuts.
In taking these actions, there are rules for making
decisions; we call these “decision rules”. We’re going
to say in this simple example that the way people
decide which action to take when they get sick is by
flipping a coin: heads they go to their family doctor;
tails they go to the ER.”
initial
In the first session, a concept model was used
to teach agent-based modeling concepts to participants
and to provide experience and comfort with the
participatory process. This concept model was a
highly simplified and obviously flawed model intended
to induce active participant engagement. The scenario
modeled and the resulting boundary object are
reproduced in the Results section. First, though, we
describe the diagramming conventions which we
taught using this first boundary object. The core
modeling team began developing these conventions
prior to the first session in an attempt to capture the
fundamental components needed to specify agent
actions and interactions.
The conventions are
described in Table 1.
Figure 1a re-creates the initial drawing based on
the above vignette. After describing and drawing the
scenario for the concept model, we then demonstrated
how it could be represented in a computer model
(programmed in NetLogo). The model interface
(Figure 1b) was designed ahead of the meeting to
resemble the boundary object drawing as it would be
initially drawn. As each element of the model
interface was introduced, its parallel in the drawn
boundary object was indicated on the white board.
Next, we asked participants what was wrong with
the model and what needed to be added. We then
edited the drawing, adhering to the conventions.
Figure 1c shows the final drawing reflecting the
changes and additions suggested by participants. Some
of the suggestions from participants included addition
of specialist provider agents, a decision rule as to
source of care based on severity of signs and
symptoms, a decision rule as to whether to seek care
which took access to quality care into account, and a
system fact that too much health care may do harm.
In the second session, after reviewing boundary
object drawing conventions, we divided participants
into three small groups and asked them to use the
conventions to create a drawing which addressed the
question “how can primary care make individuals and
2.2 Subsequent boundary objects
As the group model building sessions
progressed, additional boundary objects emerged based
on the communication needs of the group. The goal in
the creation of all visual objects used to elicit and
illustrate the model was to achieve boundary objects
that were understandable and transformable by all
participants. In the Results section below, we describe
those different boundary objects and practical advice
regarding their application.
3. Results
3.1 Boundary objects for depicting agent
decisions and interactions
We presented a “concept model” in session 1
as a device for teaching agent-based modeling concepts
to participants and for providing experience and
comfort with the participatory process.
After
2952
Table 1 – Recommended drawing conventions for agent actions and interactions
ABM component
Description
Environment
A blank white board or piece of paper
System facts
Statements written on rays of sun
Illustration/symbol
Fact
Agents
Stick figures or buildings representing
different types of people or
organizations
Agent states
Coloring/shading of agent figure
Agent decision rules
Written in caption bubbles (e.g., as
if/then statements)
Agent actions and interactions
Arrows showing impact on another
agent or on another relationship (arrow
pointing to an arrow)
Flow of information*
Arrows indicating the decision
process(es) which this information
impacts
Agent traits*
Indicated in perforated box under agent
* Not included in original conventions but added based on demonstrated need
communities healthier?”. Participants did not adhere
tightly to the drawing conventions. We believe that
this was partly due to an incomplete set of conventions.
Two elements that we quickly realized were missing
were 1) a means to convey the flow of information and
2) a way to portray agent traits. For the former, we
decided to treat information flow as an interaction
pointing to the relevant decision bubble that was
affected. For the latter, we chose to list relevant agent
traits in a box with a perforated border below the agent
(Table 1).
The broad nature of the question posed yielded
wide-ranging answers from the three small groups.
When the large group re-assembled, the core modeling
team and participants collaboratively created a
boundary object drawing which depicted one or more
phenomena mentioned by all three groups (Figure 2).
Specifically, we depicted 1) that multiple
environmental determinants influence the health of
individuals and that these determinants vary from
individual to individual, and 2) that some patients only
seek care when they are ill, while others seek regular
preventive care as well as care when they are ill. This
2953
became the basis for the initial model that would be
presented back to the group in the following session.
The ensuing two group model building sessions
consisted of iteratively demonstrating a model based
on the most recent session, experimenting live with the
model, soliciting input from participants, and
collectively deciding what changes or additions to
make next to the model. In these sessions, we found
that the group tended to feel a loss of ownership of the
model when situations arose which required the
modeling team to make decisions about model content
which had not been specified by the group via the
boundary object. While it is inevitable that previously
unforeseen assumptions will have to be made in the
course of programming a model between sessions, the
need to minimize such assumptions underscores the
importance of creating and maintaining effective
boundary objects.
3.2 Behavior-over-time graphs
1a
A mainstay boundary object used in early
development of system dynamics models is the
behavior-over-time chart [9]. Here, participants are
asked to free-hand sketch the behavior of some
phenomenon of interest over time based on their
understanding of system functioning. We found this
tool useful as a boundary object for participatory ABM
as well. For instance, when participants decided that
including different types of diseases in the model was
important, we used behavior-over-time charts to come
to consensus understandings of what the major disease
categories should be and how they would affect health
over time (Figure 3).
1b
3.3 Boundary objects for predicting aggregate
outcomes
In the fifth GMB session, we sought predictions
from participants about the health and health care
utilization of two different types of patients receiving
care in two different types of neighborhoods: patients
with high versus low tendency to access care in
neighborhoods where primary care was delivered by
either providers with whom patients had long-term
relationships or by urgent care providers
(“relationship-based” versus “episodic” care). This
was a change from asking patients to think
mechanistically about how agents behave and why, as
in previous sessions. The intent was to compare group
predictions to outputs of the model with a goal of
uncovering either unexpected aggregate behavior
emerging from the actions of individuals, or unrealistic
model assumptions. To facilitate this exercise, we
1c
2954
Figure 1. a) Re-creation of starting boundary
object used to depict scenario given in concept
model exercise, b) Corresponding NetLogo model
interface, c) Final boundary object for concept
model as modified based on participant input
modeling team), it allowed them to express some part
of their system understanding in a concrete way.
Below, we consider the extent to which this interface
itself functions as both boundary object and
“microworld” [17-19].
Figure 2. Boundary object which served as the
basis for the earliest model specified by the group
Figure 3. Consensus behavior-over-time graphs
showing health effects of four categories of
disease over time
- Multiple environmental determinants influence the
health of individuals, and these determinants vary from
individual to individual. Some patients only seek care
when they are ill, while others seek regular preventive
care as well as care when they are ill.
4. Discussion
While long used for development of system
dynamics models, participatory group model building
has not been traditionally applied to agent-based model
development. A challenge of doing so is the lack of
established boundary objects with which diverse
participants can communicate their multi-level system
understandings across disciplinary, social, or cultural
boundaries. Here, we have described the boundary
objects that evolved in the course of the group
development of an ABM of health care seeking and
delivery.
We have proposed a set of drawing
conventions for depicting agent decisions and
interactions (Table 1). In addition, we learned a
number of valuable lessons which may be
generalizable for other investigators wishing to apply
GMB to ABM:
developed another type of boundary object consisting
of a two-by-two grid reproduced in Figure 4a. For
each of the four scenarios represented by a square on
the grid, a member of the modeling team wrote
predictions from the group. Figure 4b shows the
NetLogo interface for the iteration of the model that
was used to test participant predictions. Participants
seemed to respond intuitively to the 2x2 table and
readily made the connection between the grid used to
solicit predictions and the grid displayed on the model
interface.
3.4 The model interface as boundary object
If boundary objects are artifacts which are
changeable by participants and allow participants to
communicate their understandings of systems across
boundaries of specialized knowledge and experience,
the NetLogo model interface itself might be considered
a boundary object. Because our model allowed
participants to enter their own estimated parameter
values using “sliders” (Figure 5 shows the final version
of the model created by participants and the core
x
x
2955
An early boundary object is needed for
teaching a group of non-expert stakeholders
basic ABM concepts and rules/symbology for
communicating through boundary objects
To the extent possible, boundary objects and
model interfaces should correspond in their
x
4a
x
4b
Figure 4 – a) Boundary object used to elicit
participant predictions about outcomes in four
different patient groups; b) Corresponding model
interface preserving 2x2 structure (health of
individuals represented by saturation of color of
person figure)
x
More specific questions led to more valuable
input in general and to better adherence with
boundary object conventions. We found that
even when asked a broad question, groups
tended to select a sub-question to answer that
often reflected the specific interests of a small
number of participants. When asked more
specific questions, more energy seemed to
have been spent on thinking mechanistically
about how some part of the system works. As
a concrete example, we asked participants
early on in the GMB process to sketch out
ways they felt that primary care could make
individuals and communities healthier in
small groups. Each group chose to address
some different aspect of care delivery in their
response so that, when the large group reconvened, there was little common ground to
discuss. In contrast, we asked in a later
session for participants to describe ways in
which long-term relationships between
providers and patients can improve health.
The resulting ideas emerging from small
groups were richer in the sorts of
complementary detail which enabled the
specification of mechanisms within the
model.
With hands-on exposure, the model graphical
user interface itself became an effective
boundary object.
Unlike the boundary objects used in System
Dynamics modeling, the set of boundary objects
described here allow the description of system
phenomena at both the level of the individual and the
level of aggregated groups of heterogeneous
individuals.
These boundary objects are
comprehensive in that they allow description of all the
properties, rules, actions, and interactions of agents and
environment relevant for an ABM. Yet, they are fairly
simple to explain and demonstrate for any group.
These boundary objects could prove useful in any
design process where it is necessary to elicit ideas
about a system from an individual agent perspective.
More generally, some of the above insights may be
useful for design processes requiring ongoing input
from stakeholders with different specialized
knowledge. One or more easily and universally
understandable artifacts for eliciting tacit knowledge
(i.e. boundary object) can help participants maintain a
sense of ownership in the process and product. For
this to occur, investments must be made by the
facilitators. These investments include spending time
to teach participants how to use the artifact(s) and
physical layout. Doing so seems to minimize
the cognitive barriers to participants’ linking
the consensus mental models that have been
created by the group to the actual simulation
models generated.
A feeling of group ownership of the model is
vital. The group tended to feel a loss of
ownership of the model when situations arose
which required the modeling team to make
decisions about model content which had not
been specified by the group via the boundary
object. Group ownership was restored in our
process by the core modeling team being
willing to redo major aspects of the model
based on subsequent group input, and by
participants gaining hands-on experience with
running experiments using the model.
2956
from tools which can themselves serve as boundary
objects in the model development process. For
instance, a workspace where icons corresponding to
symbols such as those in Table 1 could be “drug and
dropped” to generate code for simple models of agent
actions and interactions could be used live in a group
setting to experiment with ideas in real time.
Alternatively, such a tool could be used in a distributed
group setting for participants to asynchronously “vote”
on how they feel a system operates using a constrained
process of boundary object modification. The resulting
range of conceptual models would realistically reflect
the heterogeneous worldviews of a broad group of
participants. Still another example could be a utility
allowing the creation of functions or distributions
based on consensus-generated behavior-over-time
graphs.
demonstrating a willingness to truly allow participants
to drive the process via the artifact(s).
4.1 The model interface as microworld and
boundary object
Any successful systems model should represent a
“microworld” in that it realistically recreates the parts
of a system that are relevant to the problem at hand
[17-19]. We feel that an additional criterion for
success of a group model building effort is that the
model itself should serve some role as a boundary
object. As a boundary object, the finished model will
be transformable by users. The most obvious way to
achieve this is through an intuitive interface which
allows users to easily customize inputs or scenarios
and clearly visualize outputs. Such a product can
facilitate boundary-spanning communication within
and, later, outside of the modeling group. The agentbased modeling approach, with its individual-centric
orientation and explicit recognition of interactions
within a responsive environment, is a framework that
lends itself to intuitive understanding by non-modelers.
Creating models based on stakeholder understanding of
systems which are explicable and manipulable by
stakeholders has the potential to serve as a powerful
aid to dissemination and application by allowing the
conversation to continue beyond the model
development and formal analysis phases. A robust
participatory process for creating and cyclically
modifying these models fulfills the design science tenet
of artifact informing understanding and understanding
informing artifact.[16]
4.2 Implications
development
for
modeling
4.3 Limitations and further work
We have shared what we hope are generalizable
lessons from a single—and to our knowledge, the
first—experience with using a formal participatory
group model building approach to develop an agentbased model of a system involving health care delivery
system. The conventions and advice we provide will
be relevant to many, but not all, scenarios which future
modeling teams might encounter in similar efforts. We
consider this work a starting point for expanding the
scope of what can be achieved with a group model
building approach and for re-thinking how agent-based
models can be developed. We found the approach
fruitful and rewarding as did stakeholder participants.
We have no doubt that we finished with a very
different, and probably more relevant, model than we
would have constructed alone as “experts”. Much
work remains in terms of developing better ABM
boundary objects and techniques and perhaps
integrating these into software packages used in ABM.
software
The need for effective participatory models to
serve as boundary objects obviously suggests that
modeling software used should ideally contain tools
for creating user-friendly, intuitive model interfaces. A
separate implication of our suggestion that boundary
objects and model interfaces should correspond in their
physical layout is that modeling software may benefit
This work was supported by a methodology pilot grant
from the Patient Centered Outcomes Research Institute
(PCORI).
2957
Figure 5 – Interface for final agent-based model created through participatory group model building
process
10. References
1.
2.
3.
4.
5.
Auchincloss AH, Diez Roux AV: A
new tool for epidemiology: the
usefulness
of
dynamic-agent
models in understanding place
effects on health. American journal
of epidemiology 2008, 168(1):1-8.
Gilbert N: Agent-Based Models.
London: Sage Publications; 2007.
Diez Roux AV: Complex systems
thinking and current impasses in
health
disparities
research.
American journal of public health
2011, 101(9):1627-1634.
Bankes SC: Agent-based modeling:
a revolution? Proceedings of the
National Academy of Sciences of the
United States of America 2002, 99
Suppl 3:7199-7200.
Epstein J: Generative Social
Science: Studies in Agent-Based
6.
7.
8.
2958
Computational
Modeling.
Princeton, NJ: Princeton University
Press; 2007.
Homer JB, Hirsch GB: System
dynamics modeling for public
health:
background
and
opportunities. American journal of
public health 2006, 96(3):452-458.
Parunak HVD, Savit R, Riolo RL:
Agent-based
modeling
vs.
equation-based modeling: a case
study and users’ guide. In:
Proceedings of Multi-agent Systems
and Agent-based Simulation. 1998.
Rose J, Riolo RL, Hovmand P,
Cherng S, Ferrer R, Katendahl D,
Jaen CR, Hower T, Ruhe MC,
.Aungst H, Diez-Roux A, Stange
KC: Modeling the Paradox of
Primary Care. In: Handbook on
Systems and Complexity in Health.
9.
10.
11.
12.
13.
14.
15.
Edited by Sturmberg JP, Martin C.
New York: Springer; 2013.
Black LJ: When visuals are
boundary objects in system
dynamics work. System Dynamics
Review 2013, 29(2):70-86.
Black LJ, Andersen DF: Using
Visual
Representations
as
Boundary Objects to Resolve
Conflict in Collaborative ModelBuilding Approaches. Syst Res
Behav Sci 2012, 29(2):194-208.
Carlile PR: A pragmatic view of
knowledge
and
boundaries:
Boundary objects in new product
development. Organ Sci 2002,
13(4):442-455.
Fong A, Valerdi R, Srinivasan J:
Boundary objects as frameworks
to understand the role of system
integrators.
Systems
Research
Forum 2007, 2(1):11-18.
Borshchev A, Filippov A: From
system dynamics and discrete
event to agent based modeling:
reasons, techniques, tools. In: 22nd
International Conference of the
System Dynamics Society: July 25-29
2004; Oxford, England; 2004.
Grimm V, Berger U, Bastiansen F,
Eliassen S, Ginot V, Giske J, GossCustard J, Grand T, Heinz SK, Huse
G, Huth A, Jepsen JU, Jorgensen C,
Mooij WM, Muller B, Pe'er G, Piou
C, Railsback SF, Robbins AM,
Robbins MM, Rossmanith E, Ruger
N, Strand E, Souissi S, Stillman RA,
Vabo R, Visser U, DeAngelis DL: A
standard protocol for describing
individual-based and agent-based
models. Ecol Model 2006, 198(12):115-126.
Grimm V, Berger U, DeAngelis DL,
Polhill JG, Giske J, Railsback SF:
The ODD protocol A review and
16.
17.
18.
19.
2959
first update. Ecol Model 2010,
221(23):2760-2768.
Hevner AR, March ST, Park J, Ram
S: Design science in Information
Systems research. Mis Quart 2004,
28(1):75-105.
Zagonel
AA,
Rohrbaugh
J,
Richardson GP, Andersen DF: Using
simulation models to address
"what if" questions about welfare
reform. J Policy Anal Manag 2004,
23(4):890-901.
Zagonel dos Santos A: Reflecting on
Group Model Building Use to
Support Welfare Reform in New
York State. . University at Albany,
State University of New York; 2004.
Richardson GP, Andersen DF:
Systems thinking, mapping, and
modeling in group decision and
negotiation. In: Handbook of group
decision and negotiation. Edited by
Kilgour DM, Eden C. Dordrecht ;
New York: Springer; 2010: 313-324.