CLIMATE LITERACY AND ENERGY
AWARENESS NETWORK PATHWAY
SUMMARY OF A
RIGOROUS REVIEW PROCESS
CLN Webinar December 7, 2010
Outline

CLEAN project and CLEAN collection

Review Process

Results of first review round and gap analysis

Conclusions and Outlook
CLEAN Pathway Project Goals

Stewarding a collection of excellent teaching
resources on climate and energy science

Professional and Community Development

Online communities

Facilitate the Climate Literacy Network
CLEAN Pathway: Collection

500 excellent digital teaching resources addressing
climate science or energy awareness for grades 6-16

Resources scientifically and pedagogically reviewed

Annotations reflect reviewer comments

Resources aligned with
Climate Literacy: Essential Principles of Climate Science
 Energy Awareness Principles
 National Science Education Standards
 AAAS Project 2061Benchmarks for Science Literacy
 NAAEE Excellence in Environmental Edu. Guidelines for Learning

Framework for collection

Every activity included in collection has to
address either
 Climate
Literacy: Essential Principles of
Climate Science
http://www.climatescience.gov/Library/Literacy/
 Energy
Awareness Principles (newly
developed)






A. Earth System Energy
B. Human Energy Sources
C. Impacts of Energy Use
D. Energy Distribution/Costs
E. Energy Access and Equity
F. Decarbonization Challenges
http://cleanet.org/clean/literacy/energy.html
CLEAN review process


Phase 1 of CLEAN focused on teaching activities
Following phases: broadening of scope to videos,
visualizations, lab activities, background materials
etc.
Developing Review Criteria




Review criteria specific to teaching activities
Informed by NSDL and SERC guidelines, Merlot
criteria, DLESE, Climate Change collection scorecard
Review of e-learning materials requires additional
considerations (multi-media elements, navigation etc.)
Climate and energy science evolving science – faster
turn-around and need more media to convey
Developing Review Criteria



Initially long list of possible criteria: convergence in
phrasing of questions during 7 test review rounds
Test reviews: good agreement for good resources,
wide spread in answers for low quality resources
Weekly telecons for resource collectors
What is an excellent activity?
Definition Teaching Activity:
Relatively brief set of instructional materials that is
presented as a whole, where all the parts and ideas
are linked and part of the same activity
Educator should be excited to find this activity
when searching for teaching materials.
Avenues into collection

Existing resource pools found through online search

Resources suggested by public through online form

Targeted search after gap analysis

Collaboration with projects
that submit resources directly
to CLEAN team (iterations)
Informal vetting

Resource pools assigned to team of 9 collectors

Scanning of resource pools and informal triage

Informal triage:
Does this resource meet the CLEAN criteria?
Review questionnaire


Initial Vetting
Review
 Scientific
accuracy
 Pedagogic effectiveness
 Technical quality / Ease of use
→ 6-12 questions for each category, overall rating in
rubric format, comment box for annotations
→ Questions help to consider all relevant aspects for
each category and lead to overall rating
→ No quantitative, only qualitative recommendation
(low – medium – high priority)
Formal vetting step

Basic check for relevance to CLEAN project

Decision: “Will resource likely pass the review?”

Recommendation: Move on to initial review or put in
“holding tank”
Review: Scientific accuracy

Considerations for initial science review
 Solid,
current science
 Original
data cited and data from a quality source
 Attribution
 Valid
concepts
 Supporting
references
Science: Use of digitally available
scientific data to teach concept
Science: Simplified models of complex
Earth system to teach concept
Review: Pedagogic effectiveness

Considerations for pedagogic review
 Learning
objectives
 Accommodates
diverse learners (learning styles,
language, cultural diversity)
 Prerequisite
 Assessment
 Engaging
 Requires
skills and understandings
strategies
for students in subject and approach
independent/inquiry-based thinking
Pedagogy:
Inquiry-based labs or experiments
Pedagogy:
Pen-and-paper graphing exercise
Review: Technical quality/ Usability

Considerations for technical/usability review
 Ready
 Clear
for use, stands on its own
presentation of content
 Software/tools/resources
 Amount
 Offers
commonly found in classroom
of necessary guidance for students by instructor
comprehensive guide for instructor
 Digitally
available resource
Ease of use: Printable PDFs with
background info and instructions
Ease of use: Clear steps-by-step instruction
Expert science review



External expert with PhD in relevant field reviews
scientific quality and accuracy of resource
Activity already passed lower level science review
(75 % of resources that passed the CLEAN review
were rated scientifically excellent by experts)
Limitation/Challenge:
 Grade-level
appropriate science
 Difficult to find scientists with enough time
Panel review




Based on NSF-panel review system and AccessData
Workshops
Panel provides necessary range of expertise
Teams of 4 educators and scientists review each
teaching material based on prior reviews, final
decision about inclusion in collection
Comments of all reviewers are compiled into
annotation which includes teaching tips
Annotations



All reviewer comments, suggestions and tips are
combined in notes to users (annotations)
Annotation draft reviewed during review panels,
final clean-up during cataloging process
Annotations add considerable value, insight from
scientists or experienced educators
Demo on CLEAN Website
http://www.cleanet.org
Numbers first review cycle





Informal vetting of ~5000 teaching activities
~200 resources passed first review with medium or
high priority > forwarded to second review
142 resources passed second review with medium
or high rating > forwarded to panel review
94 resources passed panel review (18 passed on to
editorial board)
5-7 different people reviewed each resource
Gap Analysis

Holes in collection are apparent – inform targeted
search and hopefully future solicitations
Total # of
activities
# of concepts
with no match
# of concepts with
≤ 3 matches
Guiding Principle
22
2
5 of 7
EP 1
17
2
3 of 5
EP 2
20
0
3 of 6
EP 3
15
2
3 of 5
EP 4
21
1
2 of 7
EP 5
27
1
3 of 5
EP 6
8
1
5 of 5
EP 7
16
1
4 of 6
Energy Awareness
34
0
1 of 6
Conclusions

Rigorous and transparent review process

Ensures reliable and high-quality resources

Framework of Climate and Energy Literacy
Principles allow for gap analysis in collection
Outlook




Targeted search to fill gaps in collection
Broaden scope of collection to other educational
resources
Promote the collection and build a community of
educators (Professional Development/Discussions)
Refining Energy Literacy framework for collection
Detailed Review Criteria
Details about the CLEAN review process:
http://cleanet.org/clean/about/review.html
Link to Initial Vetting Questionnaire:
http://cleanet.org/files/clean/about/clean_vetting_questionnaire.pdf
Link to Review Questionnaire:
http://cleanet.org/files/clean/about/clean_review_questionnaire.pdf
Link to Expert Science Review Questionnaire:
http://cleanet.org/files/clean/about/clean_science_review_questionn.pdf
CLEAN collection: cleanet.org