Proposed Sources of Cognitive Complexity in PARCC Items and Tasks: Mathematics
August 31, 2012
Overview
In proposing sources of cognitive complexity and combining those sources into complexity indices, we
have considered PARCC’s intended goals and uses of cognitive complexity (from the June 25 PARCC
memo):
−
−
−
Provide a systematic, replicable method for determining item cognitive complexity
Provide measurement precision at all levels of the test score scales
Enable development of test forms with adequate score reliability to support achievement
growth interpretations
We considered several sources of complexity in ELA/Literacy and mathematics items—PARCC ideas,
research in educational measurement, and long standing practice in other assessment programs. We
concluded that cognitive complexity is multidimensional, which is consistent with PARCC’s view in the
June 25, 2012 memo. In addition, we considered the number of sources item writers would have to code
with each item and the additional burden that that this place on their challenging task; the degree to
which the set of proposed sources represents a comprehensive view of cognitive complexity in
ELA/Literacy and mathematics items; and whether the sources can be combined into a single complexity
index to facilitate item evaluation and selection and forms assembly. The ways in which these sources
interact with one another, and the degree to which they are related to item difficulty, is only partially
known. The details in this framework represent the best professional judgment of those who drafted it.
For both ELA/Literacy and mathematics, the cognitive complexity for polytomous items is judged for the
highest score level of the scoring rubric. The cognitive complexity for the partial credit score levels of
these items is defined by a proposed rule based system, described in the document 4. Responses to
change requests 08-31-12.pdf.
In mathematics, one role of cognitive complexity is to aid in strongly focusing the assessments where
the standards focus most closely, 70 percent or more on the major work in grades 3-8. The major work
at each grade level is sometimes the most difficult or subtle for students to master, and for teachers to
effectively implement instructional strategies related to that work. Accurately targeting this focus
through the use of cognitive complexity in the development of the items and tasks may positively
impact mathematics curricula, as well, by emphasizing deeper understanding than was typical in the
past.
A few examples that support the shift in emphasis for certain aspects of mathematical complexity are
presented below.
−
−
The National Mathematics Advisory Panel cited research showing that textbooks in the
elementary grades in China include more instances of difficult mathematics facts (e.g., 15 – 8)
than textbooks in the United States in which facts such as 5 + 3 were more common. A standard
that is intended to assess students’ facility in the recall of basic facts must include more difficult
as well as easier facts.
On the TIMSS assessment, less than one half of eighth grade students in the United States could
correctly solve a problem involving fractions such as 3/4 + 8/3 + 11/8. This may be because
Cognitive Complexity: Mathematics
Page 1
−
−
simpler problems, such as 3/4 + 1/3, that are amenable to workarounds 1, appear with greater
frequency. More challenging problems must appear on assessments with sufficient frequency so
that the ideas of fraction equivalence that underpin such problems (see 5.NF.1 in the Standards)
are appropriately addressed.
At the high school level, systems of two equations in two unknowns are often presented in the
form ax + by = c, where a, b, and c are integers, and quadratic equations such as ¼ c(c – 1) = c
are rarely encountered. The use of a variety of presentations may help students to gain
experience in applying skills and knowledge more flexibly, as college- and career-readiness
requires.
At all grade levels, students’ experiences with solving word/story problems are often limited to
problems with an arithmetic structure and are based on identifying “key words” that are then
used as hints for strategies to be applied in the solution of those problems. Assessments that
include word problems with an underlying algebraic structure broaden certain aspects of
mathematical complexity in those assessments.
In this document we recommend five individual sources to explain cognitive complexity in mathematics.
We define each source and provide a rationale for including it and identify where in the item and task
development, review, and approval process the source is recorded and reviewed. We acknowledge that
the individual sources potentially exert both main and interaction effects when they are combined to
create an overall indication on cognitive complexity. We define each individual source of complexity
holding constant (so to speak) all other sources of complexity. And we propose weights for the sources
and demonstrate how those weights would work together to produce a single complexity index.
Proposed Sources of Cognitive Complexity: Mathematics
Source
Mathematical Content
Mathematical Practices
Stimulus Material
Response Mode
Processing Demand
Item and task writers will indicate the level of cognitive complexity and proficiency level
targeted by each item. Content Developers will review these complexity levels indicated by item
and task writers. Review committees will review the Overall Index as they review items and may
refer to the intermediate indices and individual source measures as necessary.
1
See http://vimeo.com/45730600
Cognitive Complexity: Mathematics
Page 2
Mathematical Content
Explanation and Justification
The role of mathematical content in mathematical task difficulty and complexity has been addressed in
numerous studies since the 1970s (e.g., Jerman & Rees, 1972) and continues today.
At each grade level, there is a range in the level of demand in the content standards. This source of
complexity categorizes the least challenging content as low complexity and the most challenging as high
complexity, with the remainder categorized as moderate complexity. Categorizations are determined
based on typical expectations for mathematical knowledge at the grade level. New mathematical
concepts and skills that require large shifts from previously learned concepts and skills are expected to
be more complex than those that require small shifts.
In addition, the presence of certain mathematical objects (e.g., mathematical expressions, equations,
graphs) and problem structures may contribute to this source of complexity. Several examples are
presented below. (Note: A separate source of complexity, Stimulus Material, that appears later in this
document accounts for the number of pieces of stimulus material within a task or item.)
−
−
−
−
Numbers: Depending on the grade level, the types of numbers that appear may contribute to
increased overall complexity. At the high school level, the presence of complex and irrational
numbers in one problem compared to only whole numbers and simple fractions in another may
increase the overall complexity in the first problem. Similarly, at the elementary school level,
problems that contain fractions may be more complex than those containing only whole
numbers.
Expressions and Equations: The types of numbers in an expression or equation as well as the
types and number of different operations may contribute to the complexity of problems in
which expressions and equations appear.
Diagrams, graphs, or other concrete representations: Graphs that reflect more sophisticated
representations, for example multiple histograms, may contribute to greater overall complexity
than simpler graphs such as scatterplots.
Problem structures: Word problems with underlying algebraic structures may generally increase
overall complexity, compared to word problems with underlying arithmetic structures.
Low Complexity
At this level, items reflect typical expectations for mathematical knowledge at the grade level and
generally require very small or no shifts from previously learned concepts and skills. Any numbers,
expressions, and/or equations that appear should be considered in terms of the extent to which they
are contributing to the complexity of the content for that grade level. For example, at the middle school
level an item that contains irrational numbers with an underlying algebraic structure will likely not be at
low complexity, but an item that contains whole numbers with an underlying arithmetic structure may
be. In addition to evaluating the extent to which mathematical objects that appear in the item
contribute to content complexity, a judgment of low content complexity also should be based on
utilizing grade level appropriate mathematical knowledge that requires minimal shifts from previously
acquired content.
Moderate Complexity
At the moderate level, items require that students be able to access grade level appropriate
mathematical knowledge. They sometimes require a non-significant departure in how previously
Cognitive Complexity: Mathematics
Page 3
learned concepts and skills are applied and often working with one or two complex mathematical
objects (numbers, expressions, equations, diagrams, graphs) and/or problem structures that appear in
the item.
High Complexity
The most challenging content for a grade level or course appears at the high complexity level. At this
level, items require that students be able to access grade level appropriate mathematical knowledge,
routinely requiring a significant departure in how previously learned concepts and skills are applied and
often working with multiple complex mathematical objects (numbers, expressions, equations, diagrams,
graphs, and problem structures).
Mathematical Practices
Explanation and Justification
This source of complexity reflects the level of mathematical cognitive demand in items and tasks and the
level of processing of the mathematical content. It involves what students are asked to do with
mathematical content, such as engage in application and analysis of the content. The actions that
students perform on mathematical objects also contribute to Mathematical Practices complexity.
This source of complexity incorporates the standards for Mathematical Practice, attending in particular
to the practices highlighted in the PARCC assessment design (MP.4 and MP.3, 6). As was true for
Mathematical Content, complexity for this source is based on expectations of a typical student at a
grade level and the content reflected in the Standards. Students will be at various points in their learning
when they complete the PARCC assessment and may be functioning cognitively at levels above, at, or
below the indicated level for a specific item.
The following may be contributing factors to Mathematical Practices complexity.
−
−
−
−
Prompting (MP.1 – MP.8): Items or tasks that prompt the application of a particular method
are less complex than those that expect the student to bring the method to the item as a habit
of mind.
Level of Integration: Items or tasks that require the integration of knowledge and skills of
different content standards are less likely to be of low complexity than those that do not require
integration.
Modeling (MP.4): Items or tasks in which the modeling process is less structured will be more
complex than those in which the modeling process is more structured.
Explanations, justifications, and proofs (MP.3, 6): Higher complexity items and tasks utilize
minimal scaffolding in their presentation and require extended reasoning, working with multiple
representations, and the use of well-developed communication skills. Proofs are not necessarily
always examples of high complexity items; the level of complexity will depend on the demand of
the task. Also, not all applications of procedures may be low complexity. For example,
implementing certain combinations of non-iterative procedures may be more than low
complexity.
Low Complexity
Items at this level primarily involve recalling or recognizing concepts or procedures specified in the
Standards. These items often assess routine, well practiced concepts and explicitly prompt the student
Cognitive Complexity: Mathematics
Page 4
in what he or she is to do. The student may be required to apply an algorithm or work through a highly
structured modeling process either partially or in its entirety, but is not required to come up with an
original method or to provide a reasoned argument. Items at this level require students to attend to
precision in computational fluency.
Moderate Complexity
Items at this level involve more flexibility of thinking and choice among alternative methods of solution
and/or response than do those in the low complexity category. The student is expected to apply a
variety of concepts and processes from across the discipline of mathematics. For example, the student
may be asked to represent a situation in more than one way, sketch a geometric figure that satisfies
multiple conditions, show or explain his or her work, or generate an informal mathematical justification
or proof.
Items at this level require the student to begin to reason abstractly and quantitatively and use
appropriate tools strategically. The items may also involve a developing understanding of the structure
of mathematics, and appropriate mathematical modeling practices.
High Complexity
High complexity items make heavy demands on students, because students are expected to use
reasoning, planning, synthesis, analysis, judgment, and creative thought. They may be expected to
justify mathematical statements or construct a formal mathematical argument. Items at this level
usually take more time than those at other levels due to the demands of the task, not due to the
number of parts or steps. At this level, the utilization of more sophisticated modeling practices, as well
as the ability to construct complete viable mathematical arguments, and effectively critique the
reasoning of others may be characteristics of the items or tasks.
Stimulus Material
Explanation and Justification
This dimension accounts for the number of different pieces of stimulus material in an item, as well as
the role of technology tools in the item.
Low Complexity
Low complexity involves a single piece of (or no) stimulus material (e.g., table, graph, figure, etc.) OR
single online tool (generally, incremental technology)
Moderate Complexity
Moderate complexity involves two pieces of stimulus material (e.g., combinations of tables, graphs,
figures, etc.) OR single piece of stimulus material and online tool. The technology may be incremental or
transformative.
Cognitive Complexity: Mathematics
Page 5
High Complexity
High complexity involves two pieces of stimulus material with online tool(s) OR three pieces of stimulus
material with or without online tools. If technology tools are present, they are likely to be
transformative tools.
−
−
Transformative tools: Examinees must use the online tool to solve or respond to the item; they
can’t respond to the item without the technology (e.g., use an on screen ruler to make a
measurement).
Incremental tools: Technology is involved in responding, but the tools are incidental to
responding (e.g., drag and drop).
Response Mode
Explanation and Justification
The way in which examinees are required to complete assessment activities influences an item’s
cognitive complexity. We propose that, in general, selecting a response from among given choices often
is less cognitively complex than generating an original response. This difference is due in part to the
response scaffolding (i.e., response choices) in selected response items that is absent from constructed
response items. Selected response items can be highly cognitively complex due to the influence of other
sources of complexity in test items. Response Mode interacts with other sources of complexity to
influence the level of complexity: in ELA/Literacy, with Text Complexity and Command of Textual
Evidence; in Mathematics, with Mathematical Content and Processing Demands. Further, the degree to
which response choices may be easily distinguishable or highly similar can be influenced by other
sources of complexity, such as Text Complexity and Mathematical Content.
Low Complexity
Low cognitive complexity response modes in mathematics involve primarily selecting responses and
producing short responses, rather than generating more extended responses. Examples of low
complexity response modes include single or multiple selection selected response items, drag and drop
formats, use of hot spots, items that require selecting and classifying, and producing short constructed
responses (e.g., a number, a word or a few words, a mathematical expression, or a mathematical
equation).
Moderate Complexity
Moderate complexity response modes require students to work with multiple response modes within
the same item or task, including combinations of selected responses and short constructed responses.
Additional selected response formats at the moderate level may include those that require the use of a
graphing tool or an equation editor.
High Complexity
High complexity response modes require students to construct extended written responses that may
also incorporate the use of online tools such as an equation editor, graphing tool, or other online
feature that is essential to responding. High complexity also involves coordinating and organizing details
Cognitive Complexity: Mathematics
Page 6
of the response using online resources (e.g., tools, etc.) that are available. Response Mode may be
coded as high complexity for selected response items that correspond to high complexity for
Mathematical Content, Processing Demands, or Stimulus Material.
Processing Demands
Explanation and Justification
Once we apply item simplification and UDL principles and the language of the tasks, items, and prompts
have been reviewed for bias, sensitivity, editorial correctness, and so forth, some level of linguistic
demand and reading load remains. Reading load and linguistic demands in item stems, instructions for
responding to an item, and response options contribute to the cognitive complexity of items.
Linguistic demands include vocabulary choices, phrasing, and other grammatical structures. Item
development and review processes are designed to remove any such demands that are likely to be
construct irrelevant. The remaining linguistic demands may be construct relevant or at least construct
neutral. That said, linguistic demands contribute to the complexity and cognitive load of processing,
understanding, and formulating responses to test items and tasks. Research on the role of linguistic
demands in complexity and difficulty in mathematical problem solving goes back as far as 1972 (e.g.,
Jerman & Rees, 1972). In their study of mathematical problem solving and cognitive level, Days,
Wheatley, & Kulm (1979) identify linguistic demands that are related to the difficulty of mathematics
problem solving items, which they refer to as the problem’s syntax. More recently, Ferrara, Svetina,
Skucha, and Murphy (2011) and Shaftel, Belton-Kocher, Glasnapp, and Poggio (2006) have identified
linguistic demands that are related to mathematics item difficulty and discriminations indices. We
propose five linguistic demands, taken from these studies, to identify in PARCC items as sources of
complexity:
−
−
−
−
−
Ambiguous, slang, multiple meaning, and idiomatic words or phrases
Words that may be unusual or difficult and specific to English language arts (i.e., vocabulary)
Complex verbs (i.e., verb forms of three words or more), such as had been going, would have
gone
Relative pronouns, specifically, that, who, whom, whose, which (sometimes), why
Prepositional phrases
Similarly, lengthy item stems, instructions for responding to an item, and response choices also place
reading and processing demands on examinees and may give rise to additional complexity. Ferrara et al.
(2011) defined reading load and demonstrated its relationship to item difficulty and discrimination
indices for grades 3-5 mathematics items.
In research studies, linguistic demand and reading load have been identified by counting numbers of
words, prepositional phrases, and so forth. That approach is not feasible for the thousands of PARCC
items and tasks. We propose a holistic judgment approach to determining Processing Complexity. These
holistic judgments will account for the details in the Reading Load and Linguistic Demands research
frameworks.
Low Complexity
Low complexity is generally defined as a combination of low reading load and low linguistic demand.
Compared to moderate reading demand and high reading demand, low reading demand is characterized
by simple language with few words (approximately 25 words or fewer) in an item, including the item
Cognitive Complexity: Mathematics
Page 7
stem, response choices, and other directions for responding. Low complexity for this source also is
characterized by low linguistic demands, generally, and low frequencies of all five linguistic demands
(see above).
Moderate Complexity
Moderate complexity is defined as a combination of moderate reading load and moderate linguistic
demand. Moderate reading load is characterized, generally, by a range of simple to grade appropriate
language with items that are several sentences in length. Moderate complexity for this source also is
characterized by moderate linguistic demands (i.e., generally, a few instances of some of the
frequencies of all five linguistic demands; see above).
High Complexity
High complexity is defined as a combination of high reading demand and high linguistic demand. High
reading load is characterized by grade appropriate language in prompts that are generally several
sentences in length, with high linguistic demands (i.e., generally, instances of some of the frequencies of
all five linguistic demands; see above).
Complexity Sources that Were Considered and Not Included Here
Amount of Scaffolding
We interpreted this proposal from Achieve to address scaffolding within an item as, for example, when a
complicated multistep item is broken into component steps and each step is designed to elicit a scorable
examinee response. We determined that this type of scaffolding is (a) inherent in the design of an item,
and (b) determined to some degree by other complexity sources, including the proficiency level target
and Mathematical Process specified for each item.
Number of Processing Steps
Achieve proposed number of processing steps as a potential source of cognitive complexity. We believe
that this potential source of cognitive complexity is addressed in Mathematical Process. For example,
the definition for moderate complexity for Mathematical Process includes this statement:
Items at this level involve more flexibility of thinking and choice among alternative
methods of solution and/or response than do those in the low complexity category. The
student is expected to decide what to do and how to do it, synthesizing concepts and
processes from various areas of mathematics
Composite Indices and Weights and Uses of the Indices
We now propose how to weight, combine, and use each source of complexity in a composite index of
cognitive complexity. A proposal for a different approach to creating final cognitive complexity indices,
using empirical estimated decision trees, described in the document 4. Responses to change requests
08-31-12.pdf.
Cognitive Complexity: Mathematics
Page 8
Because weights like these are based on hypotheses and policy considerations, PARCC should consider
their own hypotheses and policy considerations as they consider our recommendations.
Cognitive Complexity Indices and Weights
Mathematical Content and Practices are prominent in the Common Core State Standards because they
are fundamentally important to mathematics teaching and learning. We also believe that they are the
two primary sources of cognitive complexity in mathematics assessment items and tasks and drivers of
item difficulty. The other sources of cognitive complexity are important, and relevant primarily in
assessment rather than teaching. With those considerations in mind, we propose the following cognitive
complexity indices and weights for each index in an overall cognitive complexity index.
Proposed
Weight in the
Overall Index
Rationale for the Weight
Content Complexity
30%
Practices Complexity
40%
While content and practices are
connected, factors listed under
Practices are less dictated by the
language of the Standards
themselves, yet can lead to a wide
range in the overall complexity of an
item or task.
Processing Complexity
30%
Processing Complexity plays a
significant role in item and task
complexity, as indicated by
empirical evidence.
Overall Complexity
100%
--
Proposed Index
Note. Processing Complexity combines the sources Stimulus Material, Response Mode, and
Processing Demands (all equally weighted).
Rationales for the Proposed Weights
Because this is an index of mathematical cognitive complexity, we propose that Content and Practices
should contribute the most information. Item and task cognitive complexity that is influenced by
processing demands inherent in test items and tasks together play a significant role in the overall index
and, together, are approximately equal in importance to content or practice separately.
These proposals would result in one overall measure of cognitive complexity, with five contributing
individual sources. The overall index would be generated using one of the two options described in the
Overview of this document. Each type I mathematics task (i.e., item) would be tagged with the overall
index. Similarly, each prompt (i.e., item) in type II and III mathematics tasks would be tagged with the
overall index. In addition, each type II and III task would be tagged with the overall index to aid forms
assembly.
Cognitive Complexity: Mathematics
Page 9
Uses of the Indices
We will use the Overall Index for cognitive complexity during item review processes. When questions
arise about the complexity of an item or the appropriateness of an item because of its complexity, we
will review the intermediate indices and individual sources measures. During the forms assembly
process, PARCC and its vendors can use the Overall Index to select items, evaluate individual test forms,
and to achieve parallelism of cognitive complexity across test forms, subject to the fundamental
requirement of preserving the focus, coherence, and rigor of the Common Core State Standards through
assessment.
References
Days, H.C., Wheatley, G. H., & Kulm, G. (1979). Problem structure, cognitive level, and problem-solving
performance. Journal for Research in Mathematics Education, 10, 135-146.
Ferrara, S., Svetina, D., Skucha, S., & Murphy, A. (2011). Test design with performance standards and
achievement growth in mind. Educational Measurement: Issues and Practice, 30 (4), 3-15.
Jerman, M., & Rees, R. (1972). Predicting the relative difficulty of verbal arithmetic problems.
Educational Studies in Mathematics, 13, 269–287.
Shaftel, J., Belton-Kocher, E., Glasnapp, D., & Poggio, J. (2006). The impact of language characteristics in
mathematics items on the performance of English language learners and students with
disabilities. Educational Assessment, 11(2), 105–126.
Cognitive Complexity: Mathematics
Page 10