www.culturalcognition.net The climate-science literacy measurement problem—and how to fix it Dan M. Kahan Yale University Research Supported by: Annenberg Center for Public Policy at the University of Pennsylvanalia What am I talking about? 1. The identity-knowledge confound 2. How to detect & correct it 3. Report from an ongoing research program What am I talking about? 1. The identity-knowledge confound 2. How to detect & correct it 3. Report from an ongoing research program What am I talking about? 1. The identity-knowledge confound 2. How to detect & correct it 3. Report from an ongoing research program What am I talking about? 1. The identity-knowledge confound 2. How to detect & correct it 3. Report from an ongoing research program What am I talking about? 1. The identity-knowledge confound 2. How to detect & correct it 3. Report from an ongoing research program The knowledge-identity confound . . . . The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) * The affect heuristic The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) * The affect heuristic Affect +/- Risk perception The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) * The affect heuristic perceived cost Affect +/- Risk perception perceived benefit anything else The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) * The affect heuristic perceived cost Affect +/- Risk perception perceived benefit “trust” Poortinga, W. & Pidgeon, N.F. Trust in Risk Regulation: Cause or Consequence of the Acceptability of GM Food? Risk Analysis 25, 199-209 (2005) The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) * The affect heuristic * The cultural theory of risk * Interpretive community cultural worldview Affect +/- perceived cost Risk perception perceived benefit “trust” The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) * The affect heuristic * The cultural theory of risk * Interpretive community cultural worldview perceived cost Risk perception perceived benefit “trust” The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . climate change science lit. climate change risk perception The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . happening caused by humans climate change risk perception The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . Affect +/- climate change risk perception The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . climate change risks perception Affect +/climate change “everything else” The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . climate change risks perception Affect +/- happening caused by humans The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . climate change risks perception Cultural worldviews happening caused by humans GWRISK: “How much risk do you believe global warming pose to human health, safety, or prosperity?” [0 “None at all” to 7 “Extremely high”] “How strongly do you disagree or agree with the following statements?” [1 “Strongly disagree” to 6 “strong agree”] Source: Kahan, D.M., Hank, J.-S., Tarantola, T., Silva, C. & Braman, D. Geoengineering and Climate Change Polarization: Testing a Two-Channel Model of Science Communication. Annals of the American Academy of Political and Social Science 658, 192-222 (2015). [0 “None at all” to 7 “Extremely high”] 7 6 5 4 3 2 1 0 -1.6 -1 0 1 1.6 [0 “None at all” to 7 “Extremely high”] 7 6 5 4 3 2 1 0 -1.6 -1 0 1 1.6 [0 “None at all” to 7 “Extremely high”] 7 6 r = - 0.65, p < 0.01 5 4 3 2 1 0 -1.6 -1 0 1 1.6 Beliefs on global temperature “increase in recent decades” 100% 75% 50% 25% 0% Human caused Naturally caused No warming N = 1,885. Annenberg Public Policy Center & Cultural Cognition Project. Nationally representative sample, June 2014 (YouGov). CIs are 0.95 confidence intervals for estimated general population means. The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . happening caused by humans climate change risk perception The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . 1994 2010 The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . caused by aerosols climate change risks perception Affect +/- happening caused by humans causes skin cancer The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . caused by aerosols climate change risks perception Cultural worldview happening caused by humans causes skin cancer The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . * “climate literacy” correlates with worldview * two scales, not 1: “cause” & “no cause” * worldview-“climate literacy” interaction The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . * “climate literacy” correlates with worldview * two scales, not 1: “cause” & “no cause” * worldview-“climate literacy” interaction The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . The knowledge-identity confound . . . . 1. The Slovic-Peters-Leiserowitz model (SPL) 2. “Climate change literacy” & SPL . . . What am I talking about? 1. The identity-knowledge confound 2. How to detect & correct it 3. Report from an ongoing research program How to detect it . . . . Essential tool: Item response theory How to detect it . . . . Essential tool: Item response theory 1 1 “Group 1” .9 .9 .8 .8 .7 .7 .6 .6 “Group 2” .5 “Group 1” .5 .4 .4 .3 .3 .2 .2 .1 .1 0 “Group 2” 0 -2.5 -1 0 1 2.5 1 .9 .8 .7 .6 “Group 1” .5 .4 .3 .2 “Group 2” .1 0 -2.5 -1 0 1 2.5 -2.5 -1 0 1 2.5 How to detect it . . . . Essential tool: Item response theory 1 1 “Group 1” .9 .9 .8 .8 .7 .7 .6 .6 “Group 2” .5 “Group 1” .5 .4 .4 .3 .3 .2 .2 .1 .1 0 “Group 2” 0 -2.5 -1 0 1 2.5 1 .9 .8 .7 .6 “Group 1” .5 .4 .3 .2 “Group 2” .1 0 -2.5 -1 0 1 2.5 -2.5 -1 0 1 2.5 How to detect it . . . . Essential tool: Item response theory 1 1 “Group 1” .9 .9 .8 .8 .7 .7 .6 .6 “Group 2” .5 “Group 1” .5 .4 .4 .3 .3 .2 .2 .1 .1 0 “Group 2” 0 -2.5 -1 0 1 2.5 -2.5 1 1 .9 .9 .8 .8 .7 .7 .6 1 2.5 “Group 1” .5 .4 .4 .3 .3 .2 “Group 2” .1 0 .6 “Group 1” .5 -1 0 “Group 2” .2 .1 0 -2.5 -1 0 1 2.5 -2.5 -1 0 1 2.5 How to detect it . . . . Essential tool: Item response theory 1 .9 1 < avg religiosity .9 .8 .8 .7 .7 .6 .6 .5 > avg. religiosity < avg religiosity .5 .4 .4 .3 .3 .2 .2 .1 .1 0 > avg. religiosity 0 -2.5 -1 0 1 2.5 -2.5 1 1 .9 .9 .8 .8 .7 .7 .6 .6 < avg religiosity .5 0 1 2.5 < avg religiosity > avg. religiosity > avg. religiosity .5 .4 .4 .3 .3 .2 > avg. religiosity .1 -1 0 .2 .1 0 -2.5 -1 0 1 2.5 -2.5 -1 0 1 2.5 How to detect it . . . . Essential tool: Item response theory 1 .9 1 < avg religiosity .9 .8 .8 .7 .7 .6 .6 .5 > avg. religiosity < avg religiosity .5 .4 .4 .3 .3 .2 .2 .1 .1 0 > avg. religiosity 0 -2.5 -1 0 1 2.5 -2.5 -1 0 1 2.5 There is “solid evidence” of recent global warming due “mostly” to “human activity such as burning fossil fuels.” [agree, disagree] 1 1 .9 .9 .8 .8 .7 .7 .6 .6 < avg religiosity .5 < avg religiosity .5 .4 .4 .3 .3 .2 .2 > avg. religiosity .1 0 .1 0 -2.5 -1 0 1 2.5 -2.5 -1 0 1 2.5 How to detect it . . . . Essential tool: Item response theory 1 .9 1 < avg religiosity .9 .8 .8 .7 .7 .6 .6 .5 > avg. religiosity < avg religiosity .5 .4 .4 .3 .3 .2 .2 .1 .1 0 > avg. religiosity 0 -2.5 -1 0 1 2.5 -2.5 -1 0 1 2.5 There is “solid evidence” of recent global warming due “mostly” to “human activity such as burning fossil fuels.” [agree, disagree] 1 11 .9 .9.9 .8 .8.8 .7 .7.7 .6 .5.5 .4 .4.4 .3 .3.3 .2 > avg. religiosity .1 Democrat .6.6 < avg religiosity .5 Liberal < avg religiosity 0 Conservative Republican .2.2 .1.1 00 -2.5 -1 0 1 2.5 -2.5 -2.5 -1-1 00 11 2.5 2.5 How to detect it . . . . Essential tool: Item response theory 1 .9 .8 11 Liberal < avg religiosity Democrat .9 .9 Liberal Democrat .8 .8 Conservative Republican .7 .6 .5 > avg. religiosity .7 .7 < avg religiosity .6 .6 Conservative Republican .5 .5 .4 .4 .4 .3 .3 .3 .2 .2 .2 .1 .1 .1 0 00 -2.5 -2.5 -2 -1.5 -1 -1 -.5 00 .5 11 1.5 2 2.52.5 -2.5 -2.5 -2 -1.5 -1 -1 -.5 00 .5 11 1.5 2 2.5 2.5 There is “solid evidence” of recent global warming due “mostly” to “human activity such as burning fossil fuels.” [agree, disagree] 1 1 .9 .9 .8 .8 Conservative Republican 11 Liberal < avg religiosity .9.9 Democrat .8.8 .7 .7 .7.7 .6 .6 < avg religiosity .5 .5 .4 .4 .6.6 Liberal Democrat .5.5 .4.4 .3 .3 .3.3 .2 .2 .2.2 .1 .1 .1.1 0 0 Conservative Republican 00 -2.5 -2.5 -2 -1.5 -1 -1 -.5 00 .5 11 1.5 2 2.52.5 -2.5 -2.5 -1-1 00 11 2.5 2.5 How to fix it . . . . How to fix it . . . . by unconfounding identity & knowledge! How to fix it . . . . by unconfounding identity & knowledge! How to fix it . . . . by unconfounding identity & knowledge! How to fix it . . . . by unconfounding identity & knowledge! How to fix it . . . . by unconfounding identity & knowledge! Probability of correct response There is “solid evidence” of recent global warming due “mostly” to “human activity such as burning fossil fuels.” 1 .9 .8 Liberal Democrat .7 .6 .5 .4 .3 Conservative Republican .2 .1 0 1st percentile -2.5 16th-1 percentile 50th percentile 0 percentile 84th 1 percentile 99th2.5 Ordinary Science Intelligence What am I talking about? 1. The identity-knowledge confound 2. How to detect & correct it 3. Report from an ongoing research program Report from ongoing research program . . . Report from ongoing research program . . . “Ordinary Climate Science Intelligence” battery Report from ongoing research program . . . “Ordinary Climate Science Intelligence” battery “What gas do most scientists believe causes temperatures in the atmosphere to rise? Is it [hydrogen, helium, carbon dioxide, radon]?” Report from ongoing research program . . . “Ordinary Climate Science Intelligence” battery “What gas do most scientists believe causes temperatures in the atmosphere to rise? Is it [hydrogen, helium, carbon dioxide, radon]?” “Climate scientists believe that human-caused global warming will result in flooding of many coastal regions.” [True or False] “Climate scientists believe that human-caused global warming will increase the risk of skin cancer in human beings.” [True or false] Report from ongoing research program . . . 0 0 .1 .1 .2 .2 .3 .3 .4 .4 .5 .5 .6 .6 .7 .7 .8 .8 .9 .9 1 1 OCSI: item response functions -2 bars denote 0.95 CIs -1 0 1 2 -2 -1 0 1 2 Report from ongoing research program . . . 0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 OCSI: item response functions -2 bars denote 0.95 CIs -1 0 1 2 Report from ongoing research program . . . OCSI: item response functions bars denote 0.95 CIs Report from ongoing research program . . . OSI: item response functions Probability of correct response There is “solid evidence” of recent global warming due “mostly” to “human activity such as burning fossil fuels.” 1 .9 .8 Liberal Democrat .7 .6 .5 .4 .3 Conservative Republican .2 .1 0 1st percentile -2.5 16th-1 percentile 50th percentile 0 percentile 84th 1 percentile 99th2.5 Ordinary Science Intelligence Report from ongoing research program . . . OCSI: item response functions bars denote 0.95 CIs Report from ongoing research program . . . 2 99th percentile r = 0.32, p < 0.01 1 86th percentile -1 0 50th percentile 14th percentile 1st percentile -2 Ordinary climate science intelligence OSI & OCSI: best of friends! 2 1 0 -1 -2 1st percentile 14th percentile 50th percentile 99th percentile 86th percentile science comprehension shaded area denotes 0.95 CIs Ordinary science intelligence Report from ongoing research program . . . OSI & OCSI: best of friends! > avg Left_Right 22 99th percentile r = 0.32, p < 0.01 11 86th percentile -1 -1 00 50th percentile 14th percentile 1st percentile -2 -2 Ordinary climate science intelligence < avg Left_Right 1st percentile 14th percentile 50th percentile 99th percentile 86th percentile -2-2 -1-1 00 11 22 comprehension sciencecomprehension science shaded area denotes 0.95 CIs Ordinary science intelligence Report from ongoing research program . . . 9 OCSI & positions on global warming 8 7 9 6 8 No. correct 5 7 6 4 59 3 48 2 37 26 1 15 0 04 3 2 1 0 Human caused Naturally caused No warming Positions on global warming in “past few decades” Report from ongoing research program . . . 0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 OCSI: item response functions -2 bars denote 0.95 CIs -1 0 1 2 Report from ongoing research program . . . Unconfounding What we know bars denote 0.95 CIs from Who we are Report from ongoing research program . . . “Ordinary Climate Science Intelligence” battery “Ordinary climate science intelligence” item response curves “Climate scientists believe that the increase of atmospheric carbon dioxide associated with the burning of fossil fuels will reduce photosynthesis by plants.” [True or False] 1 of correct answer probability .1 .2 .3 .4 .5 .6 .7 .8 .9 .8 .7 .6 .5 .4 .3 .2 -1 -.5 0 .5 1 1.5 0 2 -2 -1 -.5 0 .5 1 1.5 2 -2 -1 -.5 0 .5 1 1.5 2 -1 -.5 0 .5 1 1.5 -2 2 0 .5 1 1.5 2 2 -1.5 -1 -.5 0 .5 1 1.5 2 1 .1 .2 .3 .4 .5 .6 .7 .8 .9 probability of correct answer 0 -.5 1.5 “Climate scientists believe that globally averaged surface air temperatures were higher for the first decade of the twentyfirst century (2000-2009) than for the last decade of the twentieth century (1990-1999) [True or false] 1 of correct answer probability .1 .2 .3 .4 .5 .6 .7 .8 .9 1 .9 .8 .7 .6 .5 .4 .3 .2 -1 Ordinary climate science intellience 1 Ordinary climate science intellience “Climate scientists believe that here will be positive as well as negative effects from human-caused global warming.” [True or false] .1 -1.5 .5 1 -1.5 Ordinary climate science intellience 0 -2 0 of correct answer probability .1 .2 .3 .4 .5 .6 .7 .8 .9 -2 Ordinary climate science intellience “Climate scientists believe that nuclear power generation contributes to global warming” [True or false] -.5 0 of correct answer probability .1 .2 .3 .4 .5 .6 .7 .8 .9 0 0 -1.5 -1 “Climate scientists believe that human-caused global warming has increased the number and severity of hurricanes around the world in recent decades.” [True or false] 1 1 .9 .8 .7 .6 .5 .4 .3 .2 .1 -2 -1.5 Ordinary climate science intellience “Climate scientists believe that if the North Pole icecap melted as a result of human-caused global warming, global sea levels would rise.” [True or False] “Climate scientists believe that human-caused global warming will result in flooding of many coastal regions .” [True or False] probability of correct answer -1.5 Ordinary climate science intellience Ordinary climate science intellience -2 -1.5 -1 -.5 0 .5 1 1.5 Ordinary climate science intellience 2 0 -1.5 0 .1 -2 probability of correct answer “Climate scientists believe that human-caused global warming will increase the risk of skin cancer in human beings.” [True or False] 1 .9 probability of correct answer .9 .8 .7 .6 .5 .4 .3 .2 .1 0 probability of correct answer 1 “What gas do most scientists believe causes temperatures in the atmosphere to rise? Is it [hydrogen, helium, carbon dioxide, radon]?” -2 -1.5 -1 -.5 0 .5 1 1.5 Ordinary climate science intellience Figures plot the predicted probability of correctly responding to the item conditional on score on OCSI scale. Black bars denote 0.95 CIs. The figures can thus be used to assess the relative difficulty of the items and the precision with which 2 Report from ongoing research program . . . “Ordinary Climate Science Intelligence” battery “Ordinary climate science intelligence” item response curves “Climate scientists believe that the increase of atmospheric carbon dioxide associated with the burning of fossil fuels will reduce photosynthesis by plants.” [True or False] 1 of correct answer probability .1 .2 .3 .4 .5 .6 .7 .8 .9 .8 .7 .6 .5 .4 .3 .2 -1 -.5 0 .5 1 1.5 0 2 -2 -1 -.5 0 .5 1 1.5 2 -2 -1 -.5 0 .5 1 1.5 2 -1 -.5 0 .5 1 1.5 -2 2 0 .5 1 1.5 2 2 -1.5 -1 -.5 0 .5 1 1.5 2 1 .1 .2 .3 .4 .5 .6 .7 .8 .9 probability of correct answer 0 -.5 1.5 “Climate scientists believe that globally averaged surface air temperatures were higher for the first decade of the twentyfirst century (2000-2009) than for the last decade of the twentieth century (1990-1999) [True or false] 1 of correct answer probability .1 .2 .3 .4 .5 .6 .7 .8 .9 1 .9 .8 .7 .6 .5 .4 .3 .2 -1 Ordinary climate science intellience 1 Ordinary climate science intellience “Climate scientists believe that here will be positive as well as negative effects from human-caused global warming.” [True or false] .1 -1.5 .5 1 -1.5 Ordinary climate science intellience 0 -2 0 of correct answer probability .1 .2 .3 .4 .5 .6 .7 .8 .9 -2 Ordinary climate science intellience “Climate scientists believe that nuclear power generation contributes to global warming” [True or false] -.5 0 of correct answer probability .1 .2 .3 .4 .5 .6 .7 .8 .9 0 0 -1.5 -1 “Climate scientists believe that human-caused global warming has increased the number and severity of hurricanes around the world in recent decades.” [True or false] 1 1 .9 .8 .7 .6 .5 .4 .3 .2 .1 -2 -1.5 Ordinary climate science intellience “Climate scientists believe that if the North Pole icecap melted as a result of human-caused global warming, global sea levels would rise.” [True or False] “Climate scientists believe that human-caused global warming will result in flooding of many coastal regions .” [True or False] probability of correct answer -1.5 Ordinary climate science intellience Ordinary climate science intellience -2 -1.5 -1 -.5 0 .5 1 1.5 Ordinary climate science intellience 2 0 -1.5 0 .1 -2 probability of correct answer “Climate scientists believe that human-caused global warming will increase the risk of skin cancer in human beings.” [True or False] 1 .9 probability of correct answer .9 .8 .7 .6 .5 .4 .3 .2 .1 0 probability of correct answer 1 “What gas do most scientists believe causes temperatures in the atmosphere to rise? Is it [hydrogen, helium, carbon dioxide, radon]?” -2 -1.5 -1 -.5 0 .5 1 1.5 Ordinary climate science intellience Figures plot the predicted probability of correctly responding to the item conditional on score on OCSI scale. Black bars denote 0.95 CIs. The figures can thus be used to assess the relative difficulty of the items and the precision with which 2 Report from ongoing research program . . . “Ordinary Climate Science Intelligence” 2.0! * test Shi/Tobler/Visschers/Siegrist on US sample * retest best OCSI plus some rookies * use IRT to construct best unconfounded synthesis * test conjectures on OCSI & “scientific consensus” Report from ongoing research program . . . “Ordinary Climate Science Intelligence” 2.0! * test Shi/Tobler/Visschers/Siegrist on US sample * retest best OCSI plus some rookies * use IRT to construct best unconfounded synthesis * test conjectures on OCSI & “scientific consensus” Report from ongoing research program . . . “Ordinary Climate Science Intelligence” 2.0! * test Shi/Tobler/Visschers/Siegrist on US sample * retest best OCSI plus some rookies * use IRT to construct best unconfounded synthesis * test conjectures on OCSI & “scientific consensus” Report from ongoing research program . . . “Ordinary Climate Science Intelligence” 2.0! * test Shi/Tobler/Visschers/Siegrist on US sample * retest best OCSI plus some rookies * use IRT to construct best unconfounded synthesis * test conjectures on OCSI & “scientific consensus” Report from ongoing research program . . . “Ordinary Climate Science Intelligence” 2.0! * test Shi/Tobler/Visschers/Siegrist on US sample * retest best OCSI plus some rookies * use IRT to construct best unconfounded synthesis * test conjectures on OCSI & “scientific consensus” What am I talking about? 1. The identity-knowledge confound 2. How to detect & correct it 3. Report from an ongoing research program The end! New data: shame & critical reasoning! www.culturalcognition.net
© Copyright 2026 Paperzz