A Design Inquiry: Bridging Assessment and Curriculum
Frameworks to Engage Students in Science Practices
Angela Haydel DeBarger, SRI International, [email protected]
Erika Tate, bluknowledge, [email protected]
Yves Beauvineau, Culturally Responsive Science Pedagogies, [email protected]
Mingyu Feng and Patricia Schank, SRI International, [email protected], [email protected]
Tamara Heck and Michelle Williams, Michigan State University, [email protected], [email protected]
Abstract: This reflective inquiry delves into the collaborative design process of a technologybased middle school genetics unit. With a lens on science practices, we describe how bridging
assessment and curriculum design frameworks can inform decisions in an iterative co-design
process. The paper explores the role of these design frameworks in articulating learning goals
and outcomes, designing the learning experience to motivate student inquiry, and identifying
features of technology tasks and tools to elicit artifacts and evidence. A pilot study with four
teachers and 435 students revealed how to enhance elements of the design approach to
strengthen the learning goals, improve learning experiences with visualizations, and
streamline the flow of the curriculum to emphasize productive patterns of inquiry.
Implications of this integrated approach for informing the design of learning environments,
the scalability of designs, and teacher practice are discussed.
Introduction
Extensive research situates science education as a means for broadening participation in the practice and culture
of science. Researchers have designed learning environments that present science as an important part of
everyday life using relevant science dilemmas, engage learners in authentic scientific practices such as modeling
and evidence-based explanations, and incorporate technology that helps students visualize scientific phenomena
and monitor understanding (e.g., Bell, 2004; Lee, Linn, Varma & Liu, 2010; McNeil & Krajcik, 2011).
Despite the preponderance of evidence that suggests students learn science best through these rich
experiences, school science has been reported to decontextualize science and require less rigorous learning
performance. Limited resources, such as poor access to technology, too few professional development
opportunities, and policies that endorse fragmented, low-level content standards, have challenged teachers’
ability to offer coherent learning experiences for all students (Kali, Linn, & Roseman; 2008; Lee et al., 2010).
Well-designed, coherent materials have the potential to address the aforementioned challenges. With a
lens on science practices, this paper describes how bridging assessment and curriculum design frameworks can
inform key decisions in the iterative design process of a technology-based middle school genetics curriculum.
We report emergent design knowledge aimed to help designers and researchers leverage resources in the
(re)design, study, and scale of science learning environments that are accessible to schools and promote deep
and meaningful science learning and teaching.
Theoretical Approach
Two frameworks guide our approach: Knowledge Integration and Evidence Centered-Design.
Knowledge Integration (KI)
Knowledge Integration (KI) offers a learning perspective and design framework to explain how students
develop a deep understanding of science in everyday life and designed learning environments. KI recognizes
that students maintain a repertoire of ideas about scientific topics, practices, and disciplines (Bransford, Brown,
& Cocking, 2000; Linn, Davis, & Bell, 2004). As students learn through a variety of experiences in and outside
school, they revise their repertoire by adding new ideas or changing the relation among new and existing ideas.
The KI design framework, comprised of principles, processes and patterns, guides the design of
coherent science instruction, which provides students multiple opportunities to consider all their ideas, add
scientifically sound new ideas, develop relations among these ideas, and promote an integrated understanding
(Kali, Linn, & Roseman, 2008; Lee et al., 2010; Linn & Eylon, 2006). The principles (make science accessible,
make thinking visible, help students learn from each other, and promote autonomous and lifelong learning)
communicate the nature and quality of learning experiences that promote knowledge integration. The process
(elicit ideas - add ideas - distinguish ideas - sort ideas) and patterns (e.g., predict, observe, create a model,
explain) help designers coordinate learning activities that develop a deep understanding of science.
ICLS 2014 Proceedings
1022
© ISLS
Evidence-Centered Design (ECD)
Evidence-centered design (ECD; Mislevy & Haertel, 2006) provides tools for creating valid assessments within
learning environments. ECD involves an analysis of the substantive domain; the construction of an assessment
argument; specification of tasks, rubrics and psychometric models; and the implementation and delivery of tasks
within an operational assessment. A critical contribution of ECD is its provision of processes and structures to
describe epistemic practices within disciplines. ECD facilitates the articulation of (1) learning goals (knowledge,
practices, abilities and the integration thereof), (2) evidence produced in the form of actions by and among
students, and (3) features of environments to elicit the desired evidence and learning goals. By making the
underlying evidentiary argument for an assessment explicit, ECD facilitates coherence in assessment design. As
technologies (e.g., visualizations and simulations) become further integrated in learning environments, ECD is
critical for communicating decisions among the experts involved the design of these assessments.
Coordinating and Integrating KI and ECD
Our design approach aims to attend carefully to three dimensions: (1) learning goals that integrate core ideas,
practices, and cross-cutting concepts within the discipline of science; (2) clear articulation of evidence, artifacts,
and the ways students should engage to produce these; and (3) the features and flow of activity designs and
participation structures. See Table 1. While KI prioritizes deepening students’ understanding in a discipline
through practice, ECD heightens awareness of assessment design within learning environments—how we know
learning is taking place. Along these lines, KI and ECD function at different grain sizes in the design space.
Table 1: Mapping of KI and ECD Theoretical Approaches to Design Dimensions for Learning Environments.
Design Dimensions
Knowledge
Integration (KI)
EvidenceCentered Design
(ECD)
Learning Goals
Evidence, Artifacts and
Engagement
Features and Flow of
Activity and Task Designs
Focus on connections
between key ideas that
promote lifelong learning
Learning as engagement
through eliciting, adding,
distinguishing, sorting ideas
Application of design
patterns to promote
knowledge integration
within and across activities
Knowledge, practices, and
abilities and combinations
thereof that will be the
target of assessments
Clear articulation of evidence
produced by students to
indicate progress toward and
attainment of learning goals
Specification of activity and
task design features to elicit
desired evidence
We openly explored how KI and ECD approaches inform the design of a genetics unit, specifically
addressing two questions: (1) What is the role of KI and ECD in articulating learning goals and outcomes? (2)
How can KI and ECD help to define the learning experience and artifacts and elicit evidence of learning?
Project Context
The project context is the development and testing of a 5-week technology-based middle school genetics unit.
Our goal is to develop students’ understanding of inheritance through science practices of constructing models
and explanations (see NRC, 2012). “How can you use genetics to feed the world in 2052?” is the driving
question. The unit introduces a situation where the world will be running short of food and fossil fuels in 2052.
Students inquire about how to selectively breed for more nutritious rice and higher endurance horses. Student
pairs complete 10 activities with frequent opportunities for whole class discussion facilitated by the teacher
using the open-source WISE 4.0 platform. Notable advantages of the WISE platform include tools such as Idea
Manager, which supports sorting ideas and constructing explanations (McElhaney et al., 2012), and WISE
Draw, a tool that students can use to draw models that illustrate a mechanism or process.
Initial Design Approach and Implementation
Our co-design process involved experts in curriculum design, assessment design, science content, science
teaching and software design. We engaged teachers as co-designers, and these teachers served as mentor
teachers to their colleagues during implementation of the unit. A key structure in facilitating conversation about
flow was the unit template, which documents for each activity (1) learning goals addressed, (2) anticipated
student problematic ideas, (3) learning experiences needed to address problematic ideas, (4) assessment
opportunities, and (4) the KI design pattern sequence to motivate student inquiry in the activity.
ICLS 2014 Proceedings
1023
© ISLS
Articulation of the Learning Goals and Outcomes
The design team referred to the Framework for K-12 Science Education (NRC, 2012) to identify disciplinary
core ideas that comprise a deep understanding of inheritance for seventh grade students. The Framework also
helped to unpack two science practices targeted by the unit: Developing and Using Models and Constructing
Explanations. Here, we followed the ECD framework to specify abilities targeted for each practice as
Knowledge Skills, and Abilities (KSAs; e.g., Ability to construct a causal explanation; Ability to construct a
model and use the model to explain a phenomenon). The KI perspective helped establish 12 measurable learning
goals that integrated content. The learning goals began with a broad statement about the core idea and science
practice (e.g., Students will be able to generate explanations that link the macro and micro
structures/processes/functions relating to genetic expression) with additional details about the content to be
addressed (e.g., An organism's characteristics can be expressed in different versions or variations called traits).
Defining the Learning Experience, Artifacts and Evidence
KI prompted us to think about how to scaffold students to write integrated explanations of heredity and develop
models. The design team identified WISE steps that engaged students in the knowledge integration process (i.e.,
elicit, add, distinguish, and sort ideas). For example, the Explanation Builder step guides students to distinguish
which ideas help explain the genetic expression process. Using ECD, evidence for each of the practices was
broadly specified in terms of potential observations we might expect to see if students are engaged in
constructing models or explanations (e.g., application of science concepts to reason about the phenomenon).
Task Features and Flow
The KI design principles (e.g., make science visible) and patterns (e.g., Predict-Observe-Explain [POE])
informed an activity flow to scaffold coherent explanations of heredity. To make the practices of scientists
visible, the activities require students to conduct selective breeding experiments using interactive visualizations.
WISE Reflection Notes were then used to elicit students’ predictions and post-observation explanations to make
students’ scientific ideas visible. ECD served to define specific design features of assessment opportunities as
characteristic features or variable features. We considered which features needed to be present in tasks (e.g.,
All items that prompt for a scientific explanation will include data or evidence in stimulus materials) and
documented how tasks might vary (e.g., the complexity of data/evidence).
Pilot Study
Four seventh grade science teachers implemented the unit in their classes for five weeks in Spring 2013; two
teachers were from a school in a Midwest suburban district, and two were from a Southern suburban district.
The Midwest school district student body is approximately 60% Caucasian, 18% African American, 9% Asian,
7% Hispanic, 5% Multi-racial and 1% American Indian and Pacific Islander. Twenty-five percent of the
students are on free or reduced price lunch. The student body of the school district in the South is approximately
64% African American, 28% Hispanic, and 8% Caucasian. Sixty-one percent of the students are on free or
reduced price lunch. 435 students from 19 classes participated in the study and worked in pairs. Prior to
implementation, all teachers participated in a 2-day professional development workshop to discuss the unit
learning goals, the activities and web-based tools, and the teacher interface to manage student data. Student
work was logged by WISE system. The team also conducted regular classroom observations, and mentor
teachers facilitated conversations with their colleagues during the implementation of the unit.
Sources of Data to Inform Revisions
Student work on assessments provided evidence of their thinking and reasoning. Memos from classroom
observations and conversations with teachers provided insight into the student and teacher reactions to particular
activities and factors that hinder implementation.
Findings
An analysis of student work revealed that on average, groups submitted four to five ideas while viewing
visualizations and selected between three to seven ideas to support explanations. One fourth (24.7%) of the
ideas were problematic and non-normative. For example, students struggled to link ideas about macro-level
observed traits and micro-level cellular-level processes. Observations revealed that teachers’ facilitation of
prediction, modeling, and explanation steps was uneven. Some teachers engaged students in productive
discourse around focal ideas in the unit (e.g., “Why do we want rice with high starch and horses with high
endurance?”), but based on the observations, there was no evidence that these rich conversations occurred
frequently. These findings demonstrated the need for additional support for teachers and students to promote
knowledge integration. Revisions to the design approach are described below.
ICLS 2014 Proceedings
1024
© ISLS
Revised Design Approach
Articulation of Focused Knowledge-In-Use Learning Goals and Evidence
We believed that students struggled to construct explanations and models that link ideas about macro-level
observations and micro-level processes in part because our learning goals needed to be much more explicit in
this regard. Using ECD, we refined the original 12 learning goals into five learning performances that more
explicitly consider targeted applications of the science practices within genetics: (1) Ability to use a model of
trait expression to explain the observed traits in an organism; (2) Ability to construct a model of processes
within a cell and use the model to explain how traits are expressed; (3) Ability to evaluate different models of
trait expression; (4) Ability to construct a scientific explanation about how sexual reproduction can affect
genetic variation; and (5) Ability to explain how sexual reproduction and expression results in trait variation.
For each learning performance, we crafted statements that clarify the evidence students would need to produce
to demonstrate proficiency. See Table 2. The KI lens with ECD helped to outline the macro (e.g., identify
observed trait) and micro (e.g., use the model to explain the mechanism of trait expression) elements.
Table 2: Example of Refined Learning Goal and Anticipated Evidence from Student Artifacts.
Learning Goal
Evidence in Student Artifacts
Ability to use a
model of trait
expression to
explain the
observed traits
in an organism
Appropriate use of visualization (model) of gene expression to explain why an organism
has a particular trait. Macro and micro elements are connected in the explanation.
● Macro: Identification the observed characteristic and trait
● Micro-structural: Identification of the location of the relevant component (s) of the cell
● Micro-functional: Description of the function of relevant component (s) of the cell
● Micro-mechanistic: Description of the mechanism of to explain trait expression
Designing a More Consistent and Coherent Learning Experience
The rearticulated learning goals and evidence statements prompted the design team to bring practices to the
forefront and further realize the design principle, make the practices of scientists more visible. We first made the
alignment of activity steps to KI patterns more transparent. Activity steps now more explicitly map on to aspects
of science practices. For example, for the POE pattern, a Brainstorm step is used to elicit students’ predictions in
conjunction with the Idea Manager Tools (Idea Basket and Explanation Builder) to support students in
documenting observations and constructing explanations. Applying ECD, we refined features of the Idea Basket
steps to promote more active and focused observations of visualizations, as shown in Figure 1. Explanation
Builder question prompts elicit more directly explanations that require links between micro- and macro-level
ideas (e.g., Explain what happens inside a rice plant that makes it high nutrition and how Farmer Wilder can
check that the rice is very nutritious.) For some activities we also extended the basic POE pattern to incorporate
modeling (Predict-Observe-Model-Explain [POME] pattern). In the POME pattern, students create a dynamic
visual model using the WISE Draw tool.
Figure 1. Screenshot of Expression Visualization with Idea Basket Prompts
Conclusions and Implications
Findings from this design study demonstrate that KI and ECD are complementary design perspectives. KI offers
strong grounding around patterns to guide the flow of curricular experiences as well as a lens that focuses on
designs promoting connected conceptual understanding in models/explanations. ECD bolsters design practices
to support coherence in the rendering and application of tools to elicit intended knowledge and skills within a
curriculum and facilitates the articulation of evidence to look for in student-generated artifacts.
ICLS 2014 Proceedings
1025
© ISLS
The preliminary nature of our study limits our ability to generalize beyond the context of this unit. Yet,
our early attention to emergent design knowledge positions us for a more rigorous investigation of student
engagement and learning outcomes during a larger scale implementation study in the 2013-2014 school year.
This paper lends insights to practical approaches for designing knowledge-in-use learning environments that
engage students in important epistemological practices and incorporate valid assessments.
This approach also supports scalability and reuse of design principles. Generating design solutions not
only involved conversations among the co-design team, but also clear documentation of decisions (e.g., new KI
patterns and the unit template). These types of design documents afford the potential reuse of designs and
scalability of assessments and tasks for related purposes. By explicating these decisions using shared schemas,
other learning environments (e.g., game-based) that incorporate these science practices can subsequently be
generated more easily by designers without having to retrace decision paths.
While teacher practice is not the focus of this paper, we highlight some implications of our approaches
for teachers. We envision the curriculum and assessments as a starting point for discourse on constructing
models and explanations. Thus, successful implementations of this unit and similar learning environments
require supports for establishing norms to promote equitable participation (e.g., Hudicourt-Barnes, 2003),
eliciting student reasoning and promoting discussion (e.g., Penuel, Beauvineau, DeBarger, Moorthy, & Allison,
2012), and using assessments to provide feedback to students (e.g., Ruiz-Primo & Furtak, 2007).
References
Bell, P. (2004). The educational opportunities of contemporary controversies in science. In M. Linn, E. Davis &
P. Bell (Eds.), Internet environments for science education (pp. 233-260). Mahwah: Lawrence Erlbaum
Associates.
Bransford, J., A. Brown, & R. Cocking. (2000). How people learn: Brain, mind, experience, and school.
Washington: National Academy Press.
Hudicourt-Barnes, J. (2003). The use of argumentation in Haitian Creole science classrooms. Harvard
Educational Review, 73(1), 73-93.
Kali, Y., Linn, M., & Roseman, J. E. (2008). Designing coherent science education. New York: Teachers
College Press.
Lee, H. - S., Linn, M. C., Varma, K., & Liu, O. L. (2010). How do technology-enhanced inquiry science units
impact classroom learning? Journal of Research in Science Teaching, 47(1), 71–90.
Linn, M. C. & Eylon, B-S. (2006). Science education: Integrating views of learning and instruction. In P.A.
Alexander and P. H. Winne (Eds.), Handbook of Educational Psychology (2nd Ed., pp. 511-544).
Mahwah, NJ: Lawrence Erlbaum Associates.
McElhaney, K., Matuk, C., Miller, D., & Linn, M. (2012). Using the idea manager to promote coherent
understanding of inquiry investigations. In van Aalst, J., Thompson, K., Jacobson, M. J., & Reimann,
P. (Eds.) (2012). The Future of Learning: Proceedings of the 10th International Conference of the
Learning Sciences (ICLS 2012) – Volume 1, Full papers. International Society of the Learning
Sciences: Sydney, NSW, AUSTRALIA.
McNeill, K. L., & Krajcik, J. S. (2011). Supporting grade 5-8 students in constructing explanations in science:
The claim, evidence, and reasoning framework for talk and writing. New Jersey: Pearson.
Mislevy, R. J., & Haertel, G. D. (2006). Implications for evidence-centered design for educational testing.
Educational Measurement: Issues and Practices, 25, 6-20.
National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts,
and core ideas. Washington, DC: The National Academies Press.
Penuel, W. R., Beauvineau, Y., DeBarger, A. H., Moorthy, S., & Allison, K. (2012). Fostering teachers' use of
talk moves to promote productive participation in scientific practices. In J. van Aalst, K. Thompson, M.
J. Jacobson & P. Reimann (Eds.), The future of learning: Proceedings of the 10th international
conference of the learning sciences (ICLS 2012) – Volume 2, short papers, symposia, and abstracts.
International Society of the Learning Sciences: Sydney, NSW, AUSTRALIA.
Ruiz-Primo, M. A., & Furtak, E. M. (2007). Exploring teachers' informal formative assessment practices and
students' understanding in the context of scientific inquiry. Journal of Research in Science Teaching,
44(1), 57-84.
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant Number DRL1119055. Any opinions, findings, and conclusions or recommendations expressed in this material are those of
the authors and do not necessarily reflect the views of the National Science Foundation.
ICLS 2014 Proceedings
1026
© ISLS