Learning through games? Evaluating the learning effect of a policy exercise on
European climate policy
Constanze Haug, Dave Huitema and Ivo Wenzler
Address for correspondence:
Constanze Haug
Institute for Environmental Studies (IVM)
Vrije Universiteit Amsterdam
De Boelelaan 1085
1081 HV Amsterdam
The Netherlands
Email: [email protected]
Telephone: 0031-20-5982927
Fax: 0031-20-5989553
1
ABSTRACT
One of the arguments for the use of simulation-gaming approaches in policy appraisal has consistently
been their potential to stimulate learning. Yet few studies seek to ascertain the learning effects of these
methods for participants in a systematic manner; on the whole, learning through policy games remains
both under-conceptualised and under-evaluated. This paper seeks to contribute to filling this gap by
developing a typology of learning effects (cognitive, relational, and normative) that can be expected
from policy games. We subsequently present a set of tools for measuring them and test our approach
on the case of a policy exercise on burden sharing in future European climate policy involving policymakers and experts.
On the basis of our measurements, we found limited evidence for learning from the policy exercise,
mostly in the cognitive and the relational domain. An interesting methodological innovation is the use of
concept maps for this sort of endeavour. Employed as pre- and post-measurements, they proved a
useful tool for tracing conceptual change through the exercise among participants. The paper concludes
with a plea for more systematic assessment of the learning effects of simulation-gaming approaches,
but also of methods for interactive policy appraisal, with a view to enabling a deeper discussion on the
benefits and limitations of these methods.
Keywords: Social learning; learning; policy games; policy exercises; evaluation; climate policy
2
1.
Introduction
Most environmental policy deals with problems that, using the famous term of Rittel and Webber [1],
are ‘wicked’. They are commonly regarded as complex, both scientifically and socially, and as involving
high degrees of uncertainty [2] [3]. As Rittel and Webber [1] contend, for wicked problems, there is
neither agreement about the problem definition nor about the possible nature of its solution. This fact
has spurned a long tradition of writing which suggests the need for participatory, interactive approaches
in environmental policy appraisal. An increasingly strong strand of literature further argues that, given
the complexity and uncertainty of environmental problems, policy-making and governance of complex
environmental problems should be conceived as a collective and more or less permanent learning
process [4] [5] [6]. Problem definitions should be examined from as many perspectives as possible, in
broad participatory processes that are structurally evaluated and redesigned. Experimenting with
different options is an inherent part of adaptive management, and any policy adopted should be flexible
and adjustable [7] [8] [9].
Scenario-based methods fit well into this logic since, if adequately designed, they allow policy-makers
and experts to develop and assess policy options in a ‘safe’ environment, potentially providing a means
of tackling the challenges of long-term planning under deep uncertainty. Policy games and policy
exercises add to this the value of emulating the interactive, social dimension of the problem at hand.
Departing from a future qualitative plot or storyline, they assign roles to stakeholders or experts (which
may be close to their own, but can also be very different), and confront them with a set of pre-defined
tasks or decisions which they need to complete under time constraints in a fictitious scenario setting.
Policy games have been frequently applied in the environmental domain, to topics ranging from
municipal urban planning up to large-scale, long-term issues of global environmental change.
One argument for the use of policy games has consistently been their potential to stimulate learning [10]
[11]. The literature points to their benefits in terms of building and improving relationships among
stakeholders and of fostering a holistic view of the problem, as well as to their suitability as a venue for
problem-solving and for stimulating thinking about the medium- to long-term future [12]. Yet while the
3
learning effects of using games and simulations in an educational environment are relatively well
studied in the literature [13] [14] [15] [16], learning through policy games in research or as an aid for
decision-making remains both under-conceptualised and under-evaluated. Published accounts of policy
games often claim that they provided a valuable learning experience to participants. However, how and
what participants have actually learned is rarely analysed in a systematic manner. The same applies to
the related method of scenario planning; a recent review by the European Environment Agency on
‘evaluative scenario literature’ found only few studies covering this domain [17]. The present article
aims to contribute to filling this gap, by presenting a framework for evaluating the learning effects of
policy games and a set of instruments for measuring them. We subsequently test our approach on the
case of a policy game on burden sharing in future European climate policy, which we designed and ran
with policy-makers and experts. Our work is therefore of relevance to both academics and practitioners,
who are involved in the design and implementation of appraisal processes and who are keen to evaluate
the resulting learning benefits as well as the limitations of these exercises in a more systematic manner.
Writings on learning in public policy often depart from three questions that Bennett and Howlett [18]
used as signposts in a review of the literature in this field: “who learns?”, “what is learned?” and “to
what effect?”. Among these, the question of “what is learned” through a policy exercise forms the focus
of the present article. With regard to “who learns”, while different groups of actors (participants, game
designers, the wider social network or the public) can potentially benefit from a policy game, the
emphasis here is on the learning of participants through such an exercise. The question “to what effect”,
i.e. whether policy learning ultimately leads to policy change, is very hard to detect in the case of a oneoff intervention such as a policy game. It therefore falls outside of the scope of this article. Another
potentially relevant question concerns the question of “how” learning is achieved. In our case, the
vehicle for this was the policy game; where appropriate, we also discuss the potential implications of
choices made in its design on the learning of participants.
The next section provides a short review of the use of simulation-gaming approaches in public policy and
of policy exercises as one specific sub-form of this method. Section 3 develops the concept of learning
through policy games, introduces a typology of three types of learning effects that might be expected
from these and presents the methods that we used to measure them. Section 4 outlines the context and
4
design of a policy exercise that we ran with stakeholders in European climate policy. Section 5 analyses
the learning effects of this exercise. Section 6 discusses the implications of our findings and presents
avenues for future research.
2.
Policy games and exercises in public policy
The origins of policy games, also referred to as simulation-gaming, free-form gaming, or politico-military
gaming or war gaming, lie in the military sphere. The origin of modern war games is traced back by some
authors as far as early 19th century Prussia [19]. Following applications in related domains like
international relations and crisis management from the 1950s onwards, policy gaming has increasingly
been applied to environmental and social policy issues since the 1970s. Policy exercises, first advocated
by Brewer [20] and further developed by Toth [21] [22], are one sub-form of policy games. They
responded to the need for adapting the method of model-based gaming from a military or business
context to the challenges of long-term policy development in complex social-ecological systems. In
Toth’s words, “at the heart of the process [of the policy exercise] are scenario writing of ‘future histories’
and scenario analysis via the interactive formulation and testing of alternative policies that respond to
challenges in the scenarios. These scenario-based activities take place in an organizational setting
reflecting the institutional features of the problem at hand” [21: 237].
The structure of policy exercises is generally less rigid than for other types of policy games. Their primary
goal is to advance thinking about unstructured, messy problems for which the set of relevant choices,
cause-effect relationships and outcomes is controversial or unclear. Thus, while participants in a policy
exercise are, like in other forms of games, tasked to make plans and decisions under a fictitious scenario,
the storyline is usually richer and more elaborate. Moreover, policy exercises rarely involve fixed scoring
systems that reward or penalize actions by participants. Policy exercises have been applied to
environmental policy, health, education and regulatory policy reform (see e.g. Duke [23] for examples).
A number of applications have focused on various aspects of climate policy, see Table 1 for an overview.
When
Organised by
Focus of the exercise
1990
Jill Jäger et al., Stockholm Environment International climate policy
5
Institute [24]
1992
Ferenc Toth, funded by the United Adaptation options in South East Asia under
Nations Environment Programme (UNEP) different climate impact scenarios
[25]
1993
Jan H. Klabbers et al., Dutch National Policy options for climate policy
Research Program on Global Air
Pollution and Climate Change (NRP) [26]
1995
Edward Parson, Institute for Applied Global climate negotiations - mitigation
Systems Analysis (IIASA) [27]
1997
Bernd Kasemir et al., EU research project Potential of venture capital for climate policy
ULYSSES [28]
2004
Kate Lonsdale et al., EU research project Responses to extreme sea level rise in the
ATLANTIS [29]
Thames region
2008
Kurt M. Campbell et al., Center for a New Security implications of future climate change
American Security [30]
for the United States
2008/2009 Andrew Jones et al., Sustainability Global climate negotiations - mitigation
Institute, United States [31]
Table 1. Past policy exercises focusing on climate change issues.
According to Parson [3], policy games draw their usefulness from combining two characteristics,
representation and deliberation. The basis for any policy game is a simplified model of reality or the
‘system of reference’, which, however, retains the main actors, relationships and cause-effect chains [3].
Policy games, like game theory and modelling exercises, thus aim at reducing complexity through
structured abstraction. To ensure that the game set-up lives up to these requirements, good practice in
the gaming discipline emphasises the importance of a collaborative approach to game design and the
need for drawing on stakeholder expertise early on in the design process [23]. At the same time, by
relying on interaction between real-life stakeholders instead of agents with predefined preferences,
policy games incorporate a strong deliberative element. Participants jointly explore the future
‘possibility space’, building up a shared understanding of key concepts and searching creatively for
solutions [11]. Therefore, in order to make full use of the potential of the method, and in line with the
participatory appraisal literature, participants in a policy exercise should ideally be diverse, both in
terms of (disciplinary) backgrounds and viewpoints on the problem at hand [32].
6
As stated above, there are strong claims in the literature about the learning potential of policy games,
which is often attributed to the intense, vivid experience that participants go through in the course of
such an exercise. Yet, what types of learning can theoretically be expected from a policy game? And to
what extent can we substantiate that this learning actually occurs among participants? This will be the
subject of the next section.
3.
Learning through policy exercises
3.1
Policy learning and social learning
There is a rich literature on learning in public policy and management which draws on a range of
disciplines and perspectives, from organisation theory to policy analysis and cognitive psychology. While
many authors conceive of learning as a primarily cognitive process – a change in thought, resulting from
new knowledge or past experience –, others, especially from the field of social psychology, emphasise
the experiential, embedded nature of learning and stress its relational aspects [33] [34] [35]. For the
purposes of this paper, we take Sabatier's definition as a starting point; he defines policy-oriented
learning as ‘relatively enduring alterations of thought or behavioural intentions that result from
experience and that are concerned with the attainment (or revision) of public policy' [36: 131]. In the
literature, an explicit link is often made between learning and resulting (policy) change; as Argyris puts it,
‘learning occurs when understanding, insight and explanations are connected with action’[37: 1079].
This yardstick, however, may be overly ambitious in the context of policy games, given the short term
and the typically small scale nature of the intervention.
Most approaches to learning further have in common that they distinguish between different degrees
and types of learning [38]. Writings on policy learning tend to emphasise its cognitive dimension, which
we define here as the acquisition of new, and the enhancement or revision of previously gained
knowledge. The public participation literature, analysing group dynamics and social learning in
stakeholder-based processes, offers a useful extension of this concept: Webler et al. [39] distinguish
between two components of social learning, ‘cognitive enhancement’ of participants (i.e. the acquisition
7
of knowledge) and ‘moral development’, focusing on the interactive, inter-personal dimension of
appraisal.
Furthermore, many authors differentiate between a technical level and one or two ‘higher’, conceptual
levels at which learning can take place [40] [41] [42] [43]. For Argyris, for instance, single-loop learning is
“when a mismatch is corrected without changing the underlying values and status quo that govern the
behaviours”, whereas double-loop learning means that the mismatch is “corrected by first changing the
underlying values and other features of the status quo” [37: 1087f]. Building on the literature and with a
view to developing a typology specifically suited to examine learning from policy games, we
conceptualise three types of learning that participants may take away from them:
(1) Cognitive learning, which refers to the acquisition of new or the improved structuring of existing
knowledge. Cognitive learning roughly corresponds to the concepts of first-order or single-loop learning
developed in the policy learning literature.
(2) Normative learning, which implies changes in the viewpoints, norms and values of participants as a
consequence of participating in the policy exercise. It is closely related to second-order or double-loop
learning, which also incorporates a strong normative dimension.
(3) Relational learning. This term is used for instance in the literature on ‘collaborative management’ [44]
and also bears close relation to Kolb’s work on experiential learning as well as Lave and Wenger’s
“communities of practice” [33] [34]. Relational learning refers to questions such as increased trust,
improved ability to cooperate and a better understanding of the mindsets and frames of other
participants.
We thus combine the more cognition-focused notions of single- and double-loop learning from the
policy and social learning literatures with the relational dimension emphasised by Webler et al. [39]. In
our view, all three types constitute equally valuable learning benefits from a policy game. Therefore, in
contrast to the concepts of single- and double-loop as used by Argyris, we do not consider normative
learning as in principle of a higher value than its cognitive counterpart. This is because we doubt
whether a revised viewpoint on a certain topic, if it is not accompanied by a deep-seated change in
8
values or paradigms, can really be seen as more valuable than a clearer understanding of the causal links
and complexity of the topic at hand.
As with other classifications and typologies, the three types of learning that we identify are certainly
interrelated. Yet as we elaborate in Section 5, they nonetheless display specific, distinct characteristics
which can and should be assessed separately if we are to come to a more sophisticated understanding
of the benefits of interactive policy appraisals. For instance, whereas a higher degree of normative
learning may indeed stimulate a greater desire and/or ability for cognitive learning, neither is a
necessary precondition for the other. In other words, it is certainly possible to absorb new facts without
changing one’s view on the subject as hand; at the same time, deliberation and persuasion may lead a
participant to re-evaluate known arguments and thus change his or her viewpoint without having
increased his or her knowledge base. Similarly, one may learn to better understand somebody else’s
viewpoint (in the sense of better comprehending his/her reasoning and underlying arguments), while
still disagreeing fundamentally on certain core assumptions. Thus, whereas the relationship between the
two parties changed as a consequence of an increased mutual understanding, normative learning need
not have taken place as core assumptions on both sides remain intact.
Finally, any assessment of learning inevitably begs the question of the ‘baseline’ – i.e. do certain lessons
from a policy game that the evaluator considers undesirable constitute not ‘learning’, but rather the
opposite – ‘unlearning’? In the literature on social learning, learning often appears to be conflated with
its desired outcome (e.g. [4]). Yet, we follow Reed et al. [45] in considering any shift in understanding
among participants that can be attributed to the policy exercise as learning. This way we can avoid
superimposing our own value judgment of ‘positive’ vs. ‘negative’ learning outcomes over the
objectively observable changes.
3.2
Instruments for evaluating learning
A key dilemma in measuring the learning effect of interactive appraisal exercises consists in the tension
between the fact that learning occurs through intense interactions in a social setting, while
measurements are in principle only possible at the level of the individual. It is probably impossible to
9
resolve this tension completely, but a combination of group-based (such as the debriefing session) and
individual assessments may go some way in addressing it. We follow Blackstock et al. [46: 732] in our
approach, who recommend a combination of recorded and reported data for this purpose, “gathered at
multiple points throughout the process”. Thus, we have taken measurements prior to the policy exercise
as well as immediately afterwards, complementing those with process observations during the
workshop. Ideally, there should be a further evaluative moment at a later stage to account for the fact
that some learning might occur only upon reflection, some time after the exercise. However, since at
that stage the specific learning effect of the exercise may be difficult to distinguish from the impact of
other experiences when using recorded measurements, we relied on self-reported data through
interviews with participants for this purpose. The lack of a control group represents a limitation to our
methodology which prevented us from controlling for the learning effect that the evaluation
instruments themselves may have on respondents (the ‘Hawthorne effect’).
Overall, we have used the following instruments:
-
Written ex ante and ex post surveys eliciting participants’ views on facts and normative
statements about burden-sharing in EU climate policy, the topic of the policy exercise. The same
approach was used by Groves et al. [47] testing views of Californian water managers on climatic
and management uncertainties, as well as by Chermack et al. [48] who sought to assess
improvements in ‘strategic conversation quality’ as a consequence of participating in scenario
workshops. Our surveys further inquired about participants’ backgrounds, motivation for
participating in the policy exercise and their perception of what they had learned from it. Twelve
persons filled in both pre and post surveys, resulting in a response rate of 54%. Excerpts of the
survey are included in the Appendix.
-
Participants were further asked to draw concept maps of ‘burden sharing in European climate
policy’ both prior to the exercise and immediately afterwards. A concept map is a structural
representation “consisting of nodes representing concepts and [...] lines denoting the relation
between a pair of nodes” [49:1]. Concept maps, also known as mind maps, and related to other
methods such as causal or cognitive maps, have been used for a variety of purposes, from
10
creative brainstorming to project management and as planning tools, but also have a history as
assessment tools, especially in higher education [49] [50]. Various methodologies have been
developed for tracing learning through concept maps, most prominently by Morine-Dershimer
[51]. In our case, while the response rate for the ex ante maps was close to 100%, the rate
afterwards was considerably lower, owing to the difficulty of motivating participants for yet
another task after a long and strenuous workshop day. In the end, this effectively resulted in
comparable ex ante and ex post concept maps from five participants. However, we obtained at
least one set from each occupational group present at the policy exercise (policy-makers,
academics, consultants, private sector).
-
Process observations and tape recordings. During the exercise, two members of the organising
team took notes on the interactions between participants, focusing mainly on the style of
interactions and on activity patterns of individual participants. The tape recordings were also
used to determine activity patterns by participants and to check the observations and notes
taken during the workshop. The recordings were made with the prior consent of all participants,
and the presence of the small tape recorders was quickly forgotten over the course of the day.
-
A debriefing session, generally considered a crucial element of scenario and simulation-gaming
exercises, allowed participants to share their experiences and to collectively discuss what they
had learned from the exercise. The session was tape-recorded and notes were taken by the
organising team.
-
Last but not least, in-depth telephone interviews with ten participants were conducted two to
four weeks after the workshop, prompting them to reflect on their experience and on the
strengths and limitations of the policy exercise method. The interviewees were selected among
those that had indicated their willingness to participate on the ex post survey, with a view to
covering all participant groups. The interview guide can be found in the Appendix.
11
Having developed a typology of learning effects that can be expected from policy games and presented
a set of tools for measuring them, we now proceed to introduce our empirical case, which focused on a
topic of European climate policy.
4.
The case: a policy exercise on burden sharing in future European climate policy
The policy exercise that we designed and organised took place in the context of a large research project
on options for future European climate change policy, ADAM.1 Over the past years, climate change has
become an increasingly high profile issue in the European Union (EU) and has given rise to significant
policy activity at various levels of governance [52] . Our policy exercise focused on one key element of
the European climate policy architecture, the challenge of distributing greenhouse gas emission
reduction efforts among EU Member States. Under the Kyoto Protocol, the European Community signed
up to a joint emission reduction objective, which subsequently needed to be translated into targets for
individual Member States. This process is known as ‘burden sharing’. More recently, it has been
relabelled into ‘effort sharing’ in EU policy discussions addressing this question for the post-2012
period.i Burden sharing requires striking a balance between reducing emissions where they are most
cost-effective while taking into account the diversity of socio-economic circumstances across the EU.
Increasing differences in wealth and capacity within an expanding Union on the one hand, and the need
for ever more stringent emission reduction commitments on the other suggest that the basic underlying
problem of how to allocate burdens is there to stay in European climate policy [53].ii Crucial questions in
the design of burden sharing arrangements relate to the base year from which emission reductions are
accounted for, the criteria on which to base the targets, as well as a variety of issues with regard to
implementation, monitoring and enforcement. At the time of the exercise, negotiations were still
ongoing on the EU Climate and Energy Package, the intended centrepiece of the EU’s efforts to cut
emissions over the period 2012-2020, which also includes a decision on ‘effort sharing’. Our goal,
however, was to look even further ahead and explore how this policy domain could evolve after 2020.
1
‘Adaptation and Mitigation Strategies Supporting Future European Climate Policies’, funded by the Sixth
European Framework Programme, Contract 018476-GOCE.
12
The main objective of the policy exercise was thus to appraise options for future EU burden sharing
arrangements in the medium term. Participants, playing the roles of senior decision-makers from EU
Member States and the European Commission, were tasked to negotiate key features of a post-2020
burden-sharing agreement in a simplified EU policy-making cycle, with a view to assessing their
feasibility and to identify associated challenges and trade-offs. Twenty-three experts from all over
Europe participated in the exercise. Most of them were mid-career professionals with long-standing
experience in climate policy, coming from backgrounds ranging from academia to policy-making and the
private sector. Based on the information they provided in the ex ante surveys, their level of expertise on
European climate policy and burden-sharing varied from medium to very high, with one of the
participants being the desk officer at the European Commission in charge of the draft decision on ‘effort
sharing’ in the EU Climate and Energy Package.
Discussions during the policy exercise were structured through ‘policy element cards’ that contained
design alternatives for the central features of such an agreement. Participants were free to amend cards
or invent new options on blank cards however they saw fit. Issues requiring a decision included the
criteria for burden sharing (e.g. emissions or GDP per capita, available emission abatement potentials,
sectoral emissions, etc.), enforcement mechanisms as well as the extent to which member states could
substitute domestic emission cuts with purchases of external carbon credits.
In order to explore how different political and economic circumstances could impact on the results of
the negotiations, participants were divided into two subgroups that went through the same steps of
play yet were confronted with two different scenarios, both set in the year 2018. The scenario in the
first group reported a high degree of international cooperation on climate change under a successful
successor agreement to the Kyoto Protocol. In the other scenario, emissions abatement was occurring
through a variety of uncoordinated bilateral and multilateral initiatives and efforts at various levels.
Remarkably from a methodological perspective, the significant differences between the two scenarios
hardly had any impact on the negotiation results in the two groups. The policy exercise did generate
interesting insights into crucial ‘deal-makers’ and ‘deal-breakers’ in the context of burden sharing,
however. These substantive outcomes are not further discussed here, but are analysed in more detail in
Stripple et al. [54].
13
Within each of the two scenario groups, participants were divided into five teams, representing senior
policy-makers of four EU member states (Germany, Poland, Spain and Sweden) and the European
Commission. We decided to include only a limited number of countries in the exercise since we wanted
to allow for deliberation inside the country teams while still having key interests and conflict lines on EU
burden sharing represented within the subset of countries we had chosen. Before the policy exercise,
participants received short role descriptions summarising important developments in their country’s
climate policy and economic outlook over the last decade. This information allowed them to deduce the
issues that had priority for them in the negotiations. In assigning the roles, care was taken to match
participants’ own backgrounds with the nationalities they were representing. Moreover, a professional
journalist published regular ‘news bulletins’ on the progress in negotiations and on positions adopted by
Parties. The policy exercise lasted for a full working day. It was preceded by a dinner on the eve of the
workshop, during which the policy exercise approach and objectives were outlined. During the dinner,
one of the architects of the previous EU burden sharing agreement - a former chief climate negotiator of
the Netherlands - gave a talk on his experience at the time, and participants had the chance to get to
know each other.
The specific learning effects of the workshop aside, participants were generally positive about their
experience at the workshop and the design of the exercise, which according to them had brought about
a very different kind of interaction. Participants stated that it had created an atmosphere “full of fun and
respect” in a very active workshop format that made it impossible for participants “to stay out of the
discussions or give very little input” and which meant that everyone was “forced to share their
expertise”. This in itself can be considered an important achievement, especially in a domain like climate
policy where an effective interface between scientists and decision-makers is surely a sine qua non for
developing policies that are effective and equitable also in the longer term. The main points of criticism
related to the limited number of countries represented in the game, which was felt to have reduced the
realistic character of the exercise, as well as to the ‘policy element’ cards. While considered useful in
principle, these were found to be too detailed and thus distractive by many. Furthermore, some
participants underscored that a more quantitative underpinning of the exercise would have increased its
relevance. Other disagreed, however, arguing that understanding a model’s assumptions and outputs
would have cost valuable workshop time and have distracted from the core of the exercise.
14
The following section will discuss the evidence for cognitive, normative and relational learning among
participants as a consequence of the policy exercise in more detail.
5.
Learning through the ADAM policy exercise
5.1
Cognitive learning
As stated earlier, we conceptualise cognitive learning as the acquisition of new, or the restructuring of
existing knowledge. In order to evaluate this dimension, we made use of the concept maps drawn by
participants before and after the policy exercise. See Figures 1 and 2 for examples of a (digitalised) ex
ante and ex post map by one participant in the policy exercise. Furthermore, we examined the
responses to questions on the self-reported learning effect in the ex post survey and in follow-up
interviews.
Figures 3 and 4. Ex ante and ex post concept map of one participant.
The evaluation of the concept maps is based on a method initially developed by Morine-Dershimer [51];
a more detailed description is contained in the Appendix. Her method relies on two principles, (1)
15
centrality – the proximity of concepts to the core of the map, which can be taken as an indicator of how
important they are in the perception of the author and (2) specificity – the extent of detail with which a
concept is worked out in sub-concepts, or subordinate branches. A comparison of shifts in the centrality
and specificity of concepts from the pre- to the post-measurements allows tracing changes in the
structuring of knowledge or a new prioritisation of certain aspects. Morine-Dershimer’s methodology is
certainly open to criticism; for example, one might argue that new ideas might rather appear on the
margins of a map than at its centre, and thus could not be captured appropriately by the
centrality/specificity measures. Yet this does not take away from its value for systematically presenting
aggregated conceptual change at the group level.iii
All items on the five available sets of concept maps were first coded based on twelve different answer
categories, developed by the research team. Subsequently, scores for the average centrality and
specificity of these categories for all ex ante and all ex post maps were calculated based on MorineDershimer’s system (see Table 3 for the results). Figure 3 and 4 provide graphical displays of mean
centrality and specificity for all concepts on the ex ante and ex post maps (n=5). The more a concept
moves to the left on the graph, the more central it has become from the ex ante to the ex post maps;
the more it moves upward, the more specific its treatment.
16
17
Figures 3 and 4. Patterns of centrality and specificity on participants’ ex ante and ex post concept maps.
Overall, conceptual shifts are clearly detectable on the concept maps. Especially issues related to the
implementation of burden sharing arrangements and to the European emissions trading system (EU ETS)
became both more central and specific on the ex post maps. This is in line with the set-up of the policy
exercise, where participants gained more in-depth knowledge on specific issues surrounding burden
sharing. In particular, workshop discussions circled extensively around the question how to design a
burden sharing agreement that is robust (containing sufficiently stringent enforcement mechanisms) yet
flexible enough to accommodate for unforeseen developments (e.g. particularly cold winters or hot
summers, which might result in emissions increases through increased heating/air-conditioning).
Similarly, over the course of the workshop, participants became increasingly aware of the strong
18
interrelation between burden sharing and the functioning of the EU ETS, which again led to these issues
becoming more central and specific on the ex post maps, whereas more general items like the actors
involved in burdensharing diminished in both centrality and specificity. Issues that stood central during
the workshop thus moved up in both evaluation categories, testifying to some degree of conceptual
change and thus cognitive learning. The size of the sample was too small to test for statistical
significance, however.
In addition, we relied on information on the cognitive learning effect of the exercise as self-reported by
participants in the ex post surveys and interviews. Several persons emphasised that the policy exercise
had provided them with a better understanding of one issue that had previously received very little
attention in the context of the negotiations on the 2008 EU Climate and Energy Package, namely the
crucial importance of the criteria on which burden sharing agreements are based. Furthermore, some
participants indicated in the interviews that the policy exercise had prompted them to reflect more
about the requirements of long-term policy-making, and about dilemmas of compliance and
enforcement. Beyond this, however, the majority agreed that the exercise had not been particularly
successful in generating new policy options and in stimulating a truly future-oriented debate.
5.2
Normative learning
Earlier, we defined normative learning as changes in viewpoints, norms and values. In order to evaluate
this dimension, we relied on pre- and post-survey questionnaires as well as the participants’ own
perceptions as reported in the ex post interviews. The pre- and post survey questionnaires included a
list of twenty propositions related to burden sharing and to European climate policy more generally,
prompting participants to indicate the extent to which they agreed or disagreed with the points made.
The list contained normative statements like “the European Commission protects the interests of the
less developed countries too much” or “the wealthier Member States care too little about economic
development of the poorer countries.” By including the same propositions in the ex ante and ex post
surveys, we attempted to measure shifts in group opinion over the course of the exercise. We did so by
comparing the standard deviations of the pre- and post-measurements. Where the standard deviation
diminishes from the pre- to the post-survey, arguably, group opinion on the statement converged over
19
the course of the exercise. By contrast, where it increases, group opinion diverges even more following
the workshop. Based on our definition of learning, we would have considered both convergence and
divergence as evidence for a change in understanding. Given our small sample size (n=12), we did not
find variation that reached the level of statistical significance, except for a few likely coincidental
comparisons. Overall, the results from the survey do not provide evidence that normative learning has
taken place. Groves et al. [47] also report rather small effects based on a similar assessment method.
The limited extent of normative shifts detected at the group level is reflected in respondents’ individual
self-assessment in the survey. Asked whether their views on burden-sharing had changed as a result of
the exercise, 44% stated that participating had “not really” changed their views on effort-sharing, 25%
responded that it had altered it “a bit”, with a further 25% responding that it had done so “to some
extent”. Along the same lines, in one of the ex post interviews, a respondent stated that the exercise
had “reaffirmed [his] view on how EU policy-making operates”. The self-reported normative learning
effect was significantly correlated with the reasons participants stated in the survey for their
participation in the policy exercise. Those whose motivation was linked to the innovative format of the
workshop and the opportunity for networking, rather than the interest in EU policy-making and climate
policy options, were also more prone to report a change in viewpoint.
Some qualifications may apply to this finding. First, the survey statements may not be the best means
for assessment as participants may tend to simply tick the same boxes as previously (given the short
time interval between the pre- and the post-measurement) instead of genuinely reconsidering the
propositions. Similarly, given the high level of expertise of the participants, almost all of whom
considered themselves very knowledgeable in this domain, there may be a certain reluctance to admit
to normative learning in their area of specialisation. Last but not least, one might hypothesise that,
other than cognitive learning, some normative learning effects are only realised much later or that they
are simply not to be expected at all from a short one-off intervention like a policy exercise. Interestingly,
however, there was one individual case where a participant’s set of pre- and post-concept maps
unexpectedly provided strong evidence for normative learning. Following an ex ante map that largely
focussed on technical design issues related to burden sharing, one of the main branches on the ex post
map read ‘remove focus from effort sharing’, which, based on our own interpretation, implies a rather
20
fundamental shift in viewpoint, namely that, whatever the details of its implementation, burden sharing
was unlikely to ever become an environmentally effective climate policy strategy.
5.3
Relational learning
Relational learning focuses on the interactive, social dimension of the policy exercise – whether
participants gained a better understanding of their counterparts’ mindsets and whether trust and ability
to cooperate in the group increased as a consequence.
Since the participants in our policy exercise were not part of an existing network (the ex post survey
indicated that the average participant only knew three other persons taking part), it was quite difficult
to trace how the group as a whole changed through the exercise. Possibly the most solid proof of
relational learning would have been the emergence of some sort of community of practice as a
consequence of the workshop. To our knowledge, however, this did not happen – and would have
indeed have been an extraordinary result from a short workshop involving international participants
from different occupational fields. It might have been easier to assess this dimension if we had made
affiliation with an existing network a selection criterion for participation.
In our case, the learning benefits therefore rather concerned other relational aspects, most notably a
better understanding of the complexity and constraints of the EU negotiation process as well as the
various ways in which the problem of burden sharing is perceived by different actors. 53% of
participants felt that the policy exercise had increased their understanding of the dynamics of EU policymaking “to some extent” or “very much”– a percentage we would consider quite high, given the
significant existing background knowledge of participants and the short duration of the exercise. This
kind of learning is also corroborated by the answers to an open question in the ex post surveys, which
prompted participants to write down three ‘key insights’ that they were taking away from the policy
exercise. Statements relating to the political process and complexities and constraints of negotiations
made up the lion share of responses to this question, as illustrated by the following remark of one
participant: “decision-making always has to take place with insufficient information. [...]Grandstanding
in negotiations may disguise basic willingness to cooperate.”
21
Given the role play character of the exercise, it is not surprising that much relational learning of
participants concerned an improved understanding of the actor or country that they were representing
in the exercise, as well as the diverging problem definitions and problem perceptions of the various
actors. As one participant stated, “you get a real sense of how difficult these negotiations are and how
you have to guess what kind of vested interest the other countries have. In my daily work I focus on
environmental benefits and you don’t try to understand so much why countries are against something
but you just think the countries is [sic] not doing their job. It makes you realise what the agenda is of
these countries and how these interests play together.” Others stressed that the policy exercise had
made them especially aware of the difficulty of the European Commission’s mandate. The Commission is
the only body to hold the right of initiative in the legislative process of the EU, yet it is the Council and
the European Parliament that pass the laws in most cases. Having experienced this relative
powerlessness in the later stages of the decision-making process, when the Commission’s aim goal
largely consists in facilitating and seeking to avoid its proposal getting ‘watered down’ too much, led one
participant to conclude that “you can bring in little as the Commission since reaching any kind of
agreement is your very first priority.”
6.
Conclusions
The purpose of this article was to introduce a framework for assessing learning through simulation
gaming approaches in policy appraisal and to test it on the example of a policy exercise on European
climate policy. Its main contribution should be seen in view of the need for a more systematic evaluation
of these methods. While claims are often made about the significant learning potential of policy games
for participants, the latter is rarely analysed in detail nor assessed in a systematic manner in the
literature. This is hardly surprising; as Duke and Geurts [23: 211] remark, in most applications, “research
efforts have to be interwoven with the uses of games in applied settings. As a consequence, most
empirical research on policy gaming is a compromise and it is often quite limited from a research-design
perspective.” We feel that the evaluation of policy games, and especially their learning benefits, does
warrant more attention than it has received – or than has been documented – to date. Our
conceptualisation of the three different types of learning through policy games, based on the literatures
22
of policy and social learning, may provide a first step towards more analytical rigour in this regard.
Conceivably, this typology, as well as the measurement tools which we introduced, may be adapted to
suit a wider ranger of applications in the domain of participatory appraisal. If learning is proclaimed to
be such a crucial goal for these kinds of activities, more care should be expended to provide evidence for
their effectiveness in achieving it.
We also described a set of instruments by which some aspects of learning through policy games can be
measured, and we subsequently applied it to the case of a policy exercise that we developed and ran.
Notwithstanding the practical difficulties, there is clearly promise in combining different sorts of
methods and data for this purpose, allowing for cross-checking and substantiation of the evaluation
results. An interesting innovation for this sort of endeavour is the use of concept maps. Employed as
pre- and post-measurements, they proved a useful tool for tracing conceptual change among
participants through the exercise. Their strength lies in the fact that they can account also for subtler
cognitive changes that participants may not necessarily be conscious of (and that would thus be missed
by self-evaluation questions in surveys or interviews), as long as these can be detected at the group level.
However, there remains a strong role for qualitative methods, especially ex post interviews, in the
evaluation of learning effects. The main weakness of our research relates to the limited number of
observations on which our measurements were based. Ultimately, sample sizes of n=12 and n=5 for sets
of matching pre- and post-surveys and pre- and post-concept maps respectively were too small to arrive
at statistically robust conclusions. Collecting more robust evaluation data will likely remain a challenge
also in future assessments of this kind as it is notoriously difficult to convince rather high-powered
participants to partake in yet another add-on to a strenuous workshop. This may equally call for further
methodological innovations that enable data collection to be embedded better into the exercise itself,
instead being perceived as a tiresome additional assignment. On a different note, we have yet to make
use of a control group, which might strengthen our findings.
How to interpret our findings in the broader context of discussions on the role of policy exercises in
policy appraisal? In certain regards, our research gives reason for optimism. However short the policy
exercise, we found proof for both cognitive and relational learning, even though the number of available
observations was too small to yield really hard evidence for the former, while evidence for the latter was
23
even weaker. The greatest benefits of the exercise were process-related, by providing insights into EU
policy-making and the art of negotiation, as well as the constraints that the various actors are operating
under. Given the urgent need for an effective interface between science and policy in the climate
change domain, this in itself may be considered justification enough for the use of the method. Two
other aspects are more worrisome, however. In terms of cognitive learning, participants found the
exercise to be of limited value in projecting them into the future and in generating new policy options,
both presumably key objectives when employing this method. Based on our observations, participants
did make use of the future setting and the news bulletins from the year 2018 as a rhetorical device in
interactions, but substantial reasoning on concrete options and strategies remained very much tied to
the present-day situation. However, this is a problem that is also encountered in the context of other
tools for strategic foresight, such as backcasting [32]. Turning to normative learning, we were not able to
detect significant changes from the pre- to the post-survey measurements, a conclusion which was
largely supported by the self-evaluation of participants.
These findings are tentative and certainly not generalisable to the simulation-gaming method as a whole.
There is no doubt that the intended function of a policy game shapes its design [55], which in turn is a
key determinant for the learning effects of participants [56]. By opting for the policy exercise method,
which typically relies on a future setting to emphasise the long-term character of policy-making, we may
have implicitly prioritised the potential for cognitive learning to the detriment of its normative
dimension. The latter might have been stimulated more in a different set-up, which does not rely on
negotiations (a feature we explicitly built into our exercise, with a view to assessing the political
feasibility of the options), but instead focuses more on building up a shared understanding among
participants. Yet if our results are confirmed by future research and the normative learning potential of
policy exercises proves limited indeed, this would be quite regrettable, given the importance of
‘reflexivity’, i.e. the willingness and ability to reconsider established goals, policy priorities and existing
knowledge in a domain like climate policy where complexity and uncertainty abound.
Future research could be directed towards a more refined assessment of relational learning. Moreover,
it may be interesting to investigate in more depth the longer-term effects of policy games, in spite of the
difficulties of distinguishing their impact from the variety of other stimuli that participants are subjected
24
to in the meantime. Last but not least, comparing the impact of different design options on the learning
effects could be interesting, as well as comparison between groups with greater and lesser expertise on
the subject of the exercise. A control test is currently under way as we are preparing to run the same
policy exercise with master students in environmental resource management; we expect the cognitive
and normative learning effects from the exercise to be greater in this second case.
25
Acknowledgments
The research was conducted under the auspices of the project Adaptation and Mitigation Strategies
(ADAM), funded by DG Research of the European Commission. The authors would like to thank Frans
Berkhout, Benjamin Pohl and Harro van Asselt for thoughtful comments on earlier drafts of this paper.
Verena Ommer and Timme van Melle provided valuable assistance in the preparation and organization
of the policy exercise. Finally, we are grateful to Ted Parson for sharing his expertise during a research
stay at the University of Michigan.
26
Appendix: Evaluation elements
Summary of Morine-Dershimer’s method for tracing conceptual change through concept maps [51]:
Based on elements identified on the concept maps, a set of categories was developed to describe
responses. Below are the twelve answer categories used for the coding of the concept maps as well as
sample responses for each of them:
Categories
Sample responses
Delineation of the burden-sharing domain
EU-15;
Emissions trading
Amount of trading; quality of certificates
Interaction
instruments
between
policies/policy Trust into EU-level measures; conditional
funding
Actors
Member states; industry
Criteria for burden-sharing
Marginal costs
GDP/capita
International context
Post-Kyoto regime; links to other markets
Adaptation to climate change
Earmarking money for adaptation; direct link to
mitigation
Politics of burden sharing
Deals between member states;
Physical climate
Temperature increase; catastrophe
Implementation
agreements
of
burden
of
emission
reduction;
sharing Compliance regime; monitoring
Temporal dimension
Path to final year; reducing emissions later on
Economic context
Oil supply; prices
Table 2. Categories and sample responses.
The organising team first coded all elements on the concept maps were first coded by assigning them to
one of the above categories. Subsequently, for each category on every map, scores for centrality and
degree of specificity were calculated. Centrality scores were calculated based on the level at which the
category first appeared on the map, relative to the map core. For example, if a concept was linked
directly to the core, its centrality score was 1. If another concept first appeared in a reference connected
to the first concept, its centrality score was 2, and so on. Specificity scores were calculated based on the
relative frequency of items associated with one category – the number of items falling under a specific
27
category was divided by the total number of items on the map. For instance, for a map with a total of 30
items, six of which were coded as belonging to the category “international context”, the degree of
specificity for this category on this map was .20.
In order to enable comparison of scores across concept maps and participants, a ‘weighting’ system was
used to obtain a measure for each participant for each concept. Thus, if a certain concept did not appear
on a map, it was assigned a specificity score of .00. The weighted measure of centrality was counted as
being two levels below the furthest level present on the map. For example, on a map that extended to
five levels, a category that had no items noted received a centrality score of seven.
Examples of normative statements included in ex ante and ex post surveys
In your view, how important are the following factors in reaching agreement on effort-sharing
in the EU?
Do not
Totally
agree at
agree
all
a. Sound and transparent economic
models with clear criteria underlying
member state targets
b. Arrangements which take account of
historic emission reductions
c. Pressure for the EU to lead
internationally
d. Strong role of the Commission as an
impartial facilitator
e. Willingness of the economically most
advanced member states to
shoulder a comparatively larger
mitigation burden
f. Willingness of the less advanced
member states to make climate
change a political priority, even if
this come at a certain economic cost
28
Examples of other questions in the ex post survey
What are the three key insights you are taking home from the REDD-ALERT policy exercise?
1. ____________________________________________________________
2. ____________________________________________________________
3. ____________________________________________________________
How useful were the following elements of the policy exercise for your personal learning
experience?
Not at all
useful
Very
useful
Scenarios
Feature cards
Discussions with participants in the
same team
Discussions with other teams
Debriefing session
Obstacles to learning - to what extent to you agree with the following statements?
Strongly
disagree
The Policy Exercise was too short
The topic of the exercise was ill-chosen
The decision situation was too
implausible/hypothetical
The scenarios were constraining
The feature cards over-structured the
discussions
Participants were not sufficiently
knowledgeable
The group was not sufficiently diverse
29
Totally
agree
Interview guide for the ex-post interviews
1. What do you see as the key advantages of the policy exercise method? And the key drawbacks?
2. Did you find the policy exercise method a useful way of thinking about the more distant future?
If so, in which way was your thinking stimulated? How could it have been stimulated even
more? If not, what was lacking in your perspective?
3. Did the ADAM policy exercise generate innovative ideas or options?
4. In how far did you feel that the PE was different from other international workshops that you
have attended? In how far did the policy exercise format influence the way of interacting among
participants?
5. In the policy exercise, you were asked to play a role. Is this the role you normally operate from?
If not, to which extent has playing the role furthered your insight in the way the actors in that
role operate in reality? If yes, was the policy exercise a realistic reflection of your daily work?
6. If we were to organise another policy exercise, what advice would you give us? What should we
do in the same way, what should we do differently?
30
References
[1]
H. W. J. Rittel and M. M. Webber, Dilemmas in a general theory of planning, Policy Sci. 4(2) (1973)
155 169.
[2]
W. C. Clark, Sustainable development of the biosphere: themes for a research program, in: W. C.
Clark and R. E. Munn (Eds.), Sustainable Development of the Biosphere, Cambridge University
Press, Cambridge, 1986.
[3]
E. A. Parson, Informing global environmental policy-making: A plea for new methods of
assessment and synthesis, Environ. Model. Assess. 2(4) (1997) 267 279.
[4]
C. Pahl-Wostl, M. Craps, A. Dewulf, E. Mostert, D. Tabara, and T. Taillieu, Social learning and water
resources management, Ecol. Soc. 12(2) (2007)
[5]
S. E. Daniels and G. B. Walker, Collaborative learning: Improving public deliberation in ecosystembased management, Environ. Impact Asses. 16(2) (1996) 71 102.
[6]
R. Ison, N. Röling, and D. Watson, Challenges to science and society in the sustainable
management and use of water: investigating the role of social learning, Environ. Sci Pol. 10(6)
(2007) 499 511.
[7]
K. N. Lee, Appraising adaptive management, Conserv. Ecol. 3(2) (1999)
[8]
D. R. Armitage, F. Berkes, and N. Doubleday, Adaptive Co-Management. Collaboration, Learning,
and Mult-Level Governance, UBC Press, Vancouver, 2007.
[9]
D. Huitema, E. Mostert, W. Egas, S. Moellenkamp, C. Pahl-Wostl, and R. Yalcin, Adaptive Water
Governance: Assessing the Institutional Prescriptions of Adaptive (Co-) Management from a
Governance Perspective and Defining a Research Agenda, Ecol. Soc. 14(1) (2009) 26
31
[10] I. Wenzler and D. Chartier, Why Do We Bother with Games and Simulations: An Organizational
Learning Perspective, Simulation Gaming 30(3) (1999) 375 384.
[11] F. Joldersma, J. L. A. Geurts, and W. J. H. van't Spijker, Spelsimulatie. Theorie, definitie en
plaatsbepaling in de bestuurskunde, Bestuurskunde 4(4) (1995) 148 156.
[12] I. Wenzler, Policy exercises : a new approach to policy development, ITS, Nijmegen, 1993.
[13] B. D. Ruben, Simulations, Games, and Experience-Based Learning: The Quest for a New Paradigm
for Teaching and Learning, Simulat. Gaming 30(4) (1999) 498 505.
[14] W. C. Kriz, Creating Effective Learning Environments and Learning Organizations through Gaming
Simulation Design, Simulat. Gaming 34(4) (2003) 495 511.
[15] R. Fortmuller, Learning through business games: Acquiring competences within virtual realities,
Simulat. Gaming (2007) 1046878107308075
[16] J. Gosen and J. Washbush, A Review of Scholarship on Assessing Experiential Learning
Effectiveness, Simulat. Gaming 35(2) (2004) 270 293.
[17] A. Volkery and T. Ribeiro, Scenario planning in public policy: Understanding use, impacts and the
role of institutional context factors, Technol. Forecast. Soc. Change 76(9) (2009) 1198 1207.
[18] C. J. Bennett and M. Howlett, The Lessons of Learning - Reconciling Theories of Policy Learning
and Policy Change, Policy Sci. 25(3) (1992) 275 294.
[19] M. Caffrey, Toward a History-Based Doctrine for Wargaming, Aerospace J. 14(3) (2000) 33 56.
[20] G. D. Brewer, Methods for synthesis: policy exercises, in: W. C. Clark and R. E. Munn (Eds.),
Sustainable development for the biosphere, Cambridge University Press, Cambridge, 1986.
32
[21] F. L. Toth, Policy Exercises: Objectives and Design Elements, Simulat. Gaming 19(3) (1988) 235 255.
[22] F. L. Toth, Policy Exercises: Procedures and Implementation, Simulat. Gaming 19(3) (1988) 256
276.
[23] R. D. Duke and J. Geurts, Policy games for strategic management, Dutch University Press,
Amsterdam, 2004.
[24] J. Jäger, N. Sonntag, D. Bernard and W. Kurz, The challenge of sustainable development in a
greenhouse world: some visions of the future. Report of a policy exercise at Bad Bleiberg, Austria,
September 2-7, 1990. Stockholm Environment Institute, Stockholm, 1990.
[25] F. L. Toth, Policy responses to climate change in Southeast Asia, in: J. Schmandt and J. Clarkson
(Eds.), The Regions and Global Warming: Impacts and Response Strategies, Oxford University
Press, New York, 1992.
[26] J. Klabbers, R. Swart, A. van Ulden, and P. Vellinga, Management of organized complexity through
gaming, in: D. Crookall and K. Arai (Eds.), Simulation and gaming across disciplines: ISAGA at a
watershed, Sage Publications, Thousand Oaks, 1995.
[27]
E.A. Parson, Global climate change policy exercise: results of a test run, Institute for Applied
Systems Analysis, Laxenburg, 1995.
[28] B. Kasemir, F. Toth, and V. Masing, Venture capital and climate policy, in: B. Kasemir, J. Jäger, C. C.
Jaeger, and M. T. Gardner (Eds.), Public Participation in Sustainability Science: A Handbook,
Cambridge University Press, Cambridge, 2003.
[29] K. G. Lonsdale, T. E. Downing, R. J. Nicholls, D. Parker, A. T. Vafeidis, R. Dawson, and J. Hall,
Plausible responses to the threat of rapid sea-level rise in the Thames Estuary, Climatic Change
91(1) (2008) 145 169.
33
[30] S. Burke and C. Parthemore, Climate change war game: major findings and background, Center for
a New American Security, Washington D.C., 2009.
[31] A. Jones, The Copenhagen climate exercise - a simulation-based role playing exercise,
Sustainability Institute, Hartland, VT, 2009.
[32] M. van de Kerkhof, Debating Climate Change, Lemma, Utrecht, 2004.
[33] D. A. Kolb, Experiential learning, Prentice Hall, Englewood Cliffs, NJ., 1984.
[34] J. Lave and E. Wenger, Situated learning: Legitimate peripheral participation, Cambridge
University Press, Cambridge, 1991.
[35] C. Blackmore, What kinds of knowledge, knowing and learning are required for addressing
resource dilemmas? A theoretical overview, Environ. Sci. Pol. 10(6) (2007) 512 525.
[36] P. A. Sabatier, An advocacy coalition framework of policy change and the role of policy-oriented
learning therein, Policy Sci. 21(2-3) (1988) 129 168.
[37] C. Argyris, A Life Full of Learning, Organ. Stud. 24(7) (2003) 1178 1192.
[38] A. Swartling and M. Nilsson, Social learning and EPI: communicative governance in swedish
climate policy formation, Ecologic, Berlin, 2007.
[39] A. Webler, H. Kastenholz, and O. Renn, Public Participation in Impact Assessment: A Social
Learning Perspective, Environ. Impact Asses. 15(5) (1995) 443 463.
[40] C. M. Fiol and M. A. Lyles, Organizational Learning, Acad. Manage. Rev. 10(4) (1985) 803 813.
[41] P. Glasbergen, Learning to manage the environment, in: J. Meadowcraft (Ed.), Democracy and the
environment: problems and prospects, Edward Elgar, Cheltenham, UK, 1996.
34
[42] P. A. Hall, Policy Paradigms, Social Learning, and the State: The Case of Economic Policymaking in
Britain, Comp. Polit. 25(3) (1993) 275 296.
[43] C. Argyris and D. A. Schön, Organizational learning. 2. Theory, method, and practice, AddisonWesley, Reading, MA, 1996.
[44] M.T. Imperial and T. Hennessey, Environmental governance in watersheds. Collaboration, public
value and accountability, Paper presented at the Twenty-First Annual Research Conference of the
Association for Public Policy Analysis and Management, 4-6 November, 1999, Washington, D.C.,
USA, 1999.
[45] M. Reed, A. C. Evely, G. Cundill, I. Fazey, J. Glass, A. Laing, J. Newig, B. Parrish, C. Prell, C. Raymond,
and L. C. Stringer, What is social learning?, Ecol. Soc. XX(YY) (2010) rZZ (online).
[46] K. L. Blackstock, G. J. Kelly, and B. L. Horsey, Developing and applying a framework to evaluate
participatory research for sustainability, Ecol. Econ. 60(4) (2007) 726 742.
[47] D. G. Groves, D. Knopman, R. Lempert, S. H. Berry, and L. Wainfan, Presenting Uncertainty About
Climate Change to Water-Resource Managers. A Summary of Workshops with the Inland Empire
Utilities Agency, RAND Corporation, Santa Monica, CA, 2008.
[48] T. J. Chermack, L. van der Merwe, and S. A. Lynham, Exploring the relationship between scenario
planning and perceptions of strategic conversation quality, Technol. Forecast. Soc. Change 74(3)
(2007) 379 390.
[49] M. A. Ruiz-Primo and R. J. Shavelson, Problems and issues in the use of concept maps in science
assessment, J. Res. Sci. Teach. 33(6) (1996)
[50] J. D. Novak, Learning, creating, and using knowledge. Concept maps as facilitative tools in schools
and corporations, L. Erlbaum Associates, Mahwah, N.J, 1998.
35
[51] G. Morine-Dershimer, Tracing Conceptual Change in Preservice Teach. Teach. Educ. 9(1) (1993)
15 26.
[52] A. Jordan, D. Huitema, H. van Asselt, T. Rayner, and F. Berkhout, Climate Change Policy in the
European Union: Confronting the Dilemmas of Adaptation and Mitigation?, Cambridge University
Press, Cambridge, UK, 2010.
[53] C. Haug and A. Jordan, Burden Sharing: Distributing Burdens or Sharing Efforts?, in: A. Jordan, D.
Huitema, H. van Asselt, T. Rayner, and F. Berkhout (Eds.), Climate Change Policy in the European
Union: Confronting the Dilemmas of Adaptation and Mitigation?, Cambridge University Press,
Cambridge, UK, 2010.
[54] J. Stripple, R. Hildingsson, A. Jordan, T. Rayner, and C. Haug, Governance choices and dilemmas in
a warmer Europe: Exploring the future, in: A. Jordan, D. Huitema, H. van Asselt, T. Rayner, and F.
Berkhout (Eds.), Climate Change Policy in the European Union: Confronting the Dilemmas of
Adaptation and Mitigation?, Cambridge University Press, Cambridge, UK, 2010.
[55] P. Bots and E. van Daalen, Functional design of games to support natural resource management
policy development, Simulat. Gaming 38(4) (2007) 512 532.
[56] R. Ison, C. Blackmore, K. Collins, and P. Furniss, Systemic environmental decision making:
designing learning systems, Kybernetes 36(9/10) (2007) 1340
36
List of tables and figures
Table 1. Past policy exercises focusing on climate change issues.
Table 2. Categories and sample responses.
Table 3. Examples of normative statements included in pre- and post-surveys.
Table 4. Examples of other questions in the ex post survey.
Figure 1. Ex ante concept map of one participant.
Figure 2. Ex post map of the same participant.
Figure 3. Measurements of centrality and specificity on participants' ex ante concept maps.
Figure 4. Measurements of centrality and specificity on participants' ex post concept maps.
37
Endnotes
i
Interestingly, in one of the two groups in our exercise, the team playing the European Commission took
this reframing even further by referring to ‘opportunity sharing’ throughout their interventions. For the
most part, however, the notion of ‘burden sharing’ was certainly the dominant framing also in our
exercise.
ii
We worked with the assumption of accepted mitigation goals for the EU as a whole. While we
acknowledge that these could be misguided, this would most likely be in the form of underestimating
the required level of emission reductions for preventing dangerous climate change rather than the
opposite. In this case, the issue of burden sharing in the EU would gain even more importance.
iii
For future applications of this method, it might be an interesting innovation to involve participants not
just in the development, but also in the evaluation of the concept maps. For instance, participants could
be asked to reflect in writing how they thought their own map had changed compared to before the
workshop. Another option would be to have participants participate in the coding of the concept maps.
While there are clear benefits to such a more ‘participatory’ approach, it begs the question as to how far
one can stretch participants’ commitment beyond their attendance and participation in the workshop.
38