Online Resource 5
Article Title: The unseen uncertainties in climate change: reviewing comprehension of an IPCC scenario graph
Journal Name: Climatic Change
Authors: Rosemarie McMahon (1), Michael Stauffacher (2), Reto Knutti (3),
(1) Institute for Atmospheric and Climate Science, ETH Zurich, Switzerland. Email: [email protected]
(2) Institute for Environmental Decisions, ETH Zurich, Switzerland. Email: [email protected]
(3) Institute for Atmospheric and Climate Science, ETH Zurich, Switzerland. Email: [email protected]
Quantitative Results
The following steps outline how the qualitative responses collected during the interviews were converted into
quantitative data and then statistically assessed.
Step 1: Interview tasks (see, Online Resource 1) were set up to measure the levels of interpretation in accordance with
the Conceptual Framework (see section 2.2 in the original article). A number of tasks were not relevant for the analysis
of interpretation levels but were important break-in questions (e.g. Task 1), rephrased questions (Task 4) and
background questions (e.g. Task 36). Of the original 38 tasks only 25 were used to assess the interpretation levels of
graph salience, scientific literacy, content knowledge, graph schema and graph comprehension.
Step 2: The 25 tasks were given quantitative score on an ordinal scale from one to five depending on the accuracy of
the response. An accurate response received a score of five whereas an inaccurate score received a score of one. The
response accuracy was assigned in accordance with the anchoring text that was defined in the performance index (see
Online Resource 2). The below example illustrates the allocation of a score to participant 16 for Task 29.
Table 1: Example of scoring allocation
For example: independent variable = Task 29: Is the IPCC prescribing something to decision makers using this graph?
Interviewee response* “Well (laughing) that depends, if it’s an index of economic prosperity, I’d pick the one that goes up. If it’s climate change or
concentration of some pollutant or something, that one (2000 constant concentration line)” [Professor in Cognitive Neuroscience, 16]
Index
5 = Most accurate e. g. Anchoring Text
1 = Least accurate e. g. Anchoring Text
e.g. “This is IPPC they don’t prescribe policy but any of the scenarios
e.g. “Instinctively I would choose the one that’s out of the rest because
could be plausible with the exception of the lowest line (2000 constant
all of them want to show me the same thing and this one is different
concentration)” [Professor in Atmospheric Science, 12]
(2000 constant concentration). Probably the guy who designed this
graph wants to emphasise this out of the rest” [Doctoral in Geometry,
15]
Result
In this example a score of 1 was allocated as the interview response*
matched the anchoring text expected for an inaccurate answer.
1
Step 3: The scores in comprehension tasks were examined to identify how accurately participants performed. It was
found that the majority scored low (0.2) on graph comprehension tasks and only climate scientists (n=3) scored high
(0.6 and 1), see below Figure 1. The novice readers (n=40) performed badly and were not able to identify the two
different types of uncertainty represented in this graph.
Figure 1: Scores in graph comprehension for all study groups
2
Step 4: The correlation between independent and dependent variables was assessed using a non-parametric test, Spearman’s rho, see results in below Table 2. A significant
correlation was found between graph comprehension and graph schema and graph salience. However, no significant correlation existed between graph comprehension and
scientific literacy and content knowledge. This suggests that low scores in graph salience could be a predictor of poor graph comprehension but having a good knowledge of
graph schemas and conventions does not guarantee better comprehension of this graph by novice readers.
Table 2 Correlation between graph comprehension and four independent variables; scientific literacy, graph salience, graph schema, and content knowledge
Spearman's rho
Graph Comprehension
Scientific
Graph
Literacy
Salience
Graph
Schema
Content
Graph
Knowledge
Comprehension
Correlation Coefficient
.301
.338*
Sig. (2-tailed)
.059
.033
.007
.148
40
40
40
40
N
.420**
.233
1.000
40
**. Correlation is significant at the 0.01 level (2-tailed).
*. Correlation is significant at the 0.05 level (2-tailed).
3
Step 5: Using the full set of participants (N=43) the differences between the study groups was assessed for graph comprehension. The results indicate that climate scientists
scored significantly higher (𝐾𝑟𝑢𝑠𝑘𝑎𝑙 𝑊𝑎𝑙𝑙𝑖𝑠 𝑡𝑒𝑠𝑡, 𝐻(3) = 10.99, 𝑝 < 0.05) than all other readers in comprehension tasks, see Table 3 below.
Table 3: Differences between scores in graph comprehension between groups
Ranks
Group
Graph Comprehension
N
1: decision makers
2: communication experts
3: academics
4: climate scientists
Total
Mean Rank
10
15.20
9
23.06
21
21.93
3
42.00
43
Test Statisticsa,b
Graph Comprehension
Chi-Square
10.985
df
3
Asymp. Sig.
Monte Carlo Sig.
.012
.007c
Sig.
99% Confidence Interval
Lower Bound
.005
Upper Bound
.009
a. Kruskal Wallis Test
b. Grouping Variable: Group
c. Based on 10000 sampled tables with starting seed 2000000.
4
Step 6: Finally the scores per group for the five interpretation levels were evaluated and the mean and standard error calculated as outlined in the below Table 4 (see also
Figure 3 in the original article). Climate scientist performed well for all interpretation levels whereas decision makers scored high on content knowledge but not in graph
comprehension. Scores in graph salience was particularly low for novice readers. Understanding of graph schemas and conventions was high among all groups while
scientific literacy was high for academics and lowest for communication experts.
Table 4 Scores for all five-interpretation levels per group
Groups
Decision
makers (n=10)
Communication
experts (n=9)
Academics
(n=21)
Climate
scientists (n=3)
Interpretation Levels
Graph Salience
M
SE
0.3
0.04
Graph Schema
M
SE
0.7
0.08
Scientific Literacy
M
SE
0.4
0.05
Content Knowledge
M
SE
0.5
0.06
Graph Comprehension
M
SE
0.2
0.01
0.2
0.03
0.7
0.05
0.3
0.04
0.3
0.06
0.2
0.01
0.3
0.03
0.7
0.04
0.5
0.04
0.3
0.03
0.3
0.01
0.6
0.06
1.0
0.00
0.7
0.12
1.0
0.00
0.9
0.12
5