Proposed Sources of Cognitive Complexity in PARCC Items and Tasks: Mathematics August 31, 2012 Overview In proposing sources of cognitive complexity and combining those sources into complexity indices, we have considered PARCC’s intended goals and uses of cognitive complexity (from the June 25 PARCC memo): − − − Provide a systematic, replicable method for determining item cognitive complexity Provide measurement precision at all levels of the test score scales Enable development of test forms with adequate score reliability to support achievement growth interpretations We considered several sources of complexity in ELA/Literacy and mathematics items—PARCC ideas, research in educational measurement, and long standing practice in other assessment programs. We concluded that cognitive complexity is multidimensional, which is consistent with PARCC’s view in the June 25, 2012 memo. In addition, we considered the number of sources item writers would have to code with each item and the additional burden that that this place on their challenging task; the degree to which the set of proposed sources represents a comprehensive view of cognitive complexity in ELA/Literacy and mathematics items; and whether the sources can be combined into a single complexity index to facilitate item evaluation and selection and forms assembly. The ways in which these sources interact with one another, and the degree to which they are related to item difficulty, is only partially known. The details in this framework represent the best professional judgment of those who drafted it. For both ELA/Literacy and mathematics, the cognitive complexity for polytomous items is judged for the highest score level of the scoring rubric. The cognitive complexity for the partial credit score levels of these items is defined by a proposed rule based system, described in the document 4. Responses to change requests 08-31-12.pdf. In mathematics, one role of cognitive complexity is to aid in strongly focusing the assessments where the standards focus most closely, 70 percent or more on the major work in grades 3-8. The major work at each grade level is sometimes the most difficult or subtle for students to master, and for teachers to effectively implement instructional strategies related to that work. Accurately targeting this focus through the use of cognitive complexity in the development of the items and tasks may positively impact mathematics curricula, as well, by emphasizing deeper understanding than was typical in the past. A few examples that support the shift in emphasis for certain aspects of mathematical complexity are presented below. − − The National Mathematics Advisory Panel cited research showing that textbooks in the elementary grades in China include more instances of difficult mathematics facts (e.g., 15 – 8) than textbooks in the United States in which facts such as 5 + 3 were more common. A standard that is intended to assess students’ facility in the recall of basic facts must include more difficult as well as easier facts. On the TIMSS assessment, less than one half of eighth grade students in the United States could correctly solve a problem involving fractions such as 3/4 + 8/3 + 11/8. This may be because Cognitive Complexity: Mathematics Page 1 − − simpler problems, such as 3/4 + 1/3, that are amenable to workarounds 1, appear with greater frequency. More challenging problems must appear on assessments with sufficient frequency so that the ideas of fraction equivalence that underpin such problems (see 5.NF.1 in the Standards) are appropriately addressed. At the high school level, systems of two equations in two unknowns are often presented in the form ax + by = c, where a, b, and c are integers, and quadratic equations such as ¼ c(c – 1) = c are rarely encountered. The use of a variety of presentations may help students to gain experience in applying skills and knowledge more flexibly, as college- and career-readiness requires. At all grade levels, students’ experiences with solving word/story problems are often limited to problems with an arithmetic structure and are based on identifying “key words” that are then used as hints for strategies to be applied in the solution of those problems. Assessments that include word problems with an underlying algebraic structure broaden certain aspects of mathematical complexity in those assessments. In this document we recommend five individual sources to explain cognitive complexity in mathematics. We define each source and provide a rationale for including it and identify where in the item and task development, review, and approval process the source is recorded and reviewed. We acknowledge that the individual sources potentially exert both main and interaction effects when they are combined to create an overall indication on cognitive complexity. We define each individual source of complexity holding constant (so to speak) all other sources of complexity. And we propose weights for the sources and demonstrate how those weights would work together to produce a single complexity index. Proposed Sources of Cognitive Complexity: Mathematics Source Mathematical Content Mathematical Practices Stimulus Material Response Mode Processing Demand Item and task writers will indicate the level of cognitive complexity and proficiency level targeted by each item. Content Developers will review these complexity levels indicated by item and task writers. Review committees will review the Overall Index as they review items and may refer to the intermediate indices and individual source measures as necessary. 1 See http://vimeo.com/45730600 Cognitive Complexity: Mathematics Page 2 Mathematical Content Explanation and Justification The role of mathematical content in mathematical task difficulty and complexity has been addressed in numerous studies since the 1970s (e.g., Jerman & Rees, 1972) and continues today. At each grade level, there is a range in the level of demand in the content standards. This source of complexity categorizes the least challenging content as low complexity and the most challenging as high complexity, with the remainder categorized as moderate complexity. Categorizations are determined based on typical expectations for mathematical knowledge at the grade level. New mathematical concepts and skills that require large shifts from previously learned concepts and skills are expected to be more complex than those that require small shifts. In addition, the presence of certain mathematical objects (e.g., mathematical expressions, equations, graphs) and problem structures may contribute to this source of complexity. Several examples are presented below. (Note: A separate source of complexity, Stimulus Material, that appears later in this document accounts for the number of pieces of stimulus material within a task or item.) − − − − Numbers: Depending on the grade level, the types of numbers that appear may contribute to increased overall complexity. At the high school level, the presence of complex and irrational numbers in one problem compared to only whole numbers and simple fractions in another may increase the overall complexity in the first problem. Similarly, at the elementary school level, problems that contain fractions may be more complex than those containing only whole numbers. Expressions and Equations: The types of numbers in an expression or equation as well as the types and number of different operations may contribute to the complexity of problems in which expressions and equations appear. Diagrams, graphs, or other concrete representations: Graphs that reflect more sophisticated representations, for example multiple histograms, may contribute to greater overall complexity than simpler graphs such as scatterplots. Problem structures: Word problems with underlying algebraic structures may generally increase overall complexity, compared to word problems with underlying arithmetic structures. Low Complexity At this level, items reflect typical expectations for mathematical knowledge at the grade level and generally require very small or no shifts from previously learned concepts and skills. Any numbers, expressions, and/or equations that appear should be considered in terms of the extent to which they are contributing to the complexity of the content for that grade level. For example, at the middle school level an item that contains irrational numbers with an underlying algebraic structure will likely not be at low complexity, but an item that contains whole numbers with an underlying arithmetic structure may be. In addition to evaluating the extent to which mathematical objects that appear in the item contribute to content complexity, a judgment of low content complexity also should be based on utilizing grade level appropriate mathematical knowledge that requires minimal shifts from previously acquired content. Moderate Complexity At the moderate level, items require that students be able to access grade level appropriate mathematical knowledge. They sometimes require a non-significant departure in how previously Cognitive Complexity: Mathematics Page 3 learned concepts and skills are applied and often working with one or two complex mathematical objects (numbers, expressions, equations, diagrams, graphs) and/or problem structures that appear in the item. High Complexity The most challenging content for a grade level or course appears at the high complexity level. At this level, items require that students be able to access grade level appropriate mathematical knowledge, routinely requiring a significant departure in how previously learned concepts and skills are applied and often working with multiple complex mathematical objects (numbers, expressions, equations, diagrams, graphs, and problem structures). Mathematical Practices Explanation and Justification This source of complexity reflects the level of mathematical cognitive demand in items and tasks and the level of processing of the mathematical content. It involves what students are asked to do with mathematical content, such as engage in application and analysis of the content. The actions that students perform on mathematical objects also contribute to Mathematical Practices complexity. This source of complexity incorporates the standards for Mathematical Practice, attending in particular to the practices highlighted in the PARCC assessment design (MP.4 and MP.3, 6). As was true for Mathematical Content, complexity for this source is based on expectations of a typical student at a grade level and the content reflected in the Standards. Students will be at various points in their learning when they complete the PARCC assessment and may be functioning cognitively at levels above, at, or below the indicated level for a specific item. The following may be contributing factors to Mathematical Practices complexity. − − − − Prompting (MP.1 – MP.8): Items or tasks that prompt the application of a particular method are less complex than those that expect the student to bring the method to the item as a habit of mind. Level of Integration: Items or tasks that require the integration of knowledge and skills of different content standards are less likely to be of low complexity than those that do not require integration. Modeling (MP.4): Items or tasks in which the modeling process is less structured will be more complex than those in which the modeling process is more structured. Explanations, justifications, and proofs (MP.3, 6): Higher complexity items and tasks utilize minimal scaffolding in their presentation and require extended reasoning, working with multiple representations, and the use of well-developed communication skills. Proofs are not necessarily always examples of high complexity items; the level of complexity will depend on the demand of the task. Also, not all applications of procedures may be low complexity. For example, implementing certain combinations of non-iterative procedures may be more than low complexity. Low Complexity Items at this level primarily involve recalling or recognizing concepts or procedures specified in the Standards. These items often assess routine, well practiced concepts and explicitly prompt the student Cognitive Complexity: Mathematics Page 4 in what he or she is to do. The student may be required to apply an algorithm or work through a highly structured modeling process either partially or in its entirety, but is not required to come up with an original method or to provide a reasoned argument. Items at this level require students to attend to precision in computational fluency. Moderate Complexity Items at this level involve more flexibility of thinking and choice among alternative methods of solution and/or response than do those in the low complexity category. The student is expected to apply a variety of concepts and processes from across the discipline of mathematics. For example, the student may be asked to represent a situation in more than one way, sketch a geometric figure that satisfies multiple conditions, show or explain his or her work, or generate an informal mathematical justification or proof. Items at this level require the student to begin to reason abstractly and quantitatively and use appropriate tools strategically. The items may also involve a developing understanding of the structure of mathematics, and appropriate mathematical modeling practices. High Complexity High complexity items make heavy demands on students, because students are expected to use reasoning, planning, synthesis, analysis, judgment, and creative thought. They may be expected to justify mathematical statements or construct a formal mathematical argument. Items at this level usually take more time than those at other levels due to the demands of the task, not due to the number of parts or steps. At this level, the utilization of more sophisticated modeling practices, as well as the ability to construct complete viable mathematical arguments, and effectively critique the reasoning of others may be characteristics of the items or tasks. Stimulus Material Explanation and Justification This dimension accounts for the number of different pieces of stimulus material in an item, as well as the role of technology tools in the item. Low Complexity Low complexity involves a single piece of (or no) stimulus material (e.g., table, graph, figure, etc.) OR single online tool (generally, incremental technology) Moderate Complexity Moderate complexity involves two pieces of stimulus material (e.g., combinations of tables, graphs, figures, etc.) OR single piece of stimulus material and online tool. The technology may be incremental or transformative. Cognitive Complexity: Mathematics Page 5 High Complexity High complexity involves two pieces of stimulus material with online tool(s) OR three pieces of stimulus material with or without online tools. If technology tools are present, they are likely to be transformative tools. − − Transformative tools: Examinees must use the online tool to solve or respond to the item; they can’t respond to the item without the technology (e.g., use an on screen ruler to make a measurement). Incremental tools: Technology is involved in responding, but the tools are incidental to responding (e.g., drag and drop). Response Mode Explanation and Justification The way in which examinees are required to complete assessment activities influences an item’s cognitive complexity. We propose that, in general, selecting a response from among given choices often is less cognitively complex than generating an original response. This difference is due in part to the response scaffolding (i.e., response choices) in selected response items that is absent from constructed response items. Selected response items can be highly cognitively complex due to the influence of other sources of complexity in test items. Response Mode interacts with other sources of complexity to influence the level of complexity: in ELA/Literacy, with Text Complexity and Command of Textual Evidence; in Mathematics, with Mathematical Content and Processing Demands. Further, the degree to which response choices may be easily distinguishable or highly similar can be influenced by other sources of complexity, such as Text Complexity and Mathematical Content. Low Complexity Low cognitive complexity response modes in mathematics involve primarily selecting responses and producing short responses, rather than generating more extended responses. Examples of low complexity response modes include single or multiple selection selected response items, drag and drop formats, use of hot spots, items that require selecting and classifying, and producing short constructed responses (e.g., a number, a word or a few words, a mathematical expression, or a mathematical equation). Moderate Complexity Moderate complexity response modes require students to work with multiple response modes within the same item or task, including combinations of selected responses and short constructed responses. Additional selected response formats at the moderate level may include those that require the use of a graphing tool or an equation editor. High Complexity High complexity response modes require students to construct extended written responses that may also incorporate the use of online tools such as an equation editor, graphing tool, or other online feature that is essential to responding. High complexity also involves coordinating and organizing details Cognitive Complexity: Mathematics Page 6 of the response using online resources (e.g., tools, etc.) that are available. Response Mode may be coded as high complexity for selected response items that correspond to high complexity for Mathematical Content, Processing Demands, or Stimulus Material. Processing Demands Explanation and Justification Once we apply item simplification and UDL principles and the language of the tasks, items, and prompts have been reviewed for bias, sensitivity, editorial correctness, and so forth, some level of linguistic demand and reading load remains. Reading load and linguistic demands in item stems, instructions for responding to an item, and response options contribute to the cognitive complexity of items. Linguistic demands include vocabulary choices, phrasing, and other grammatical structures. Item development and review processes are designed to remove any such demands that are likely to be construct irrelevant. The remaining linguistic demands may be construct relevant or at least construct neutral. That said, linguistic demands contribute to the complexity and cognitive load of processing, understanding, and formulating responses to test items and tasks. Research on the role of linguistic demands in complexity and difficulty in mathematical problem solving goes back as far as 1972 (e.g., Jerman & Rees, 1972). In their study of mathematical problem solving and cognitive level, Days, Wheatley, & Kulm (1979) identify linguistic demands that are related to the difficulty of mathematics problem solving items, which they refer to as the problem’s syntax. More recently, Ferrara, Svetina, Skucha, and Murphy (2011) and Shaftel, Belton-Kocher, Glasnapp, and Poggio (2006) have identified linguistic demands that are related to mathematics item difficulty and discriminations indices. We propose five linguistic demands, taken from these studies, to identify in PARCC items as sources of complexity: − − − − − Ambiguous, slang, multiple meaning, and idiomatic words or phrases Words that may be unusual or difficult and specific to English language arts (i.e., vocabulary) Complex verbs (i.e., verb forms of three words or more), such as had been going, would have gone Relative pronouns, specifically, that, who, whom, whose, which (sometimes), why Prepositional phrases Similarly, lengthy item stems, instructions for responding to an item, and response choices also place reading and processing demands on examinees and may give rise to additional complexity. Ferrara et al. (2011) defined reading load and demonstrated its relationship to item difficulty and discrimination indices for grades 3-5 mathematics items. In research studies, linguistic demand and reading load have been identified by counting numbers of words, prepositional phrases, and so forth. That approach is not feasible for the thousands of PARCC items and tasks. We propose a holistic judgment approach to determining Processing Complexity. These holistic judgments will account for the details in the Reading Load and Linguistic Demands research frameworks. Low Complexity Low complexity is generally defined as a combination of low reading load and low linguistic demand. Compared to moderate reading demand and high reading demand, low reading demand is characterized by simple language with few words (approximately 25 words or fewer) in an item, including the item Cognitive Complexity: Mathematics Page 7 stem, response choices, and other directions for responding. Low complexity for this source also is characterized by low linguistic demands, generally, and low frequencies of all five linguistic demands (see above). Moderate Complexity Moderate complexity is defined as a combination of moderate reading load and moderate linguistic demand. Moderate reading load is characterized, generally, by a range of simple to grade appropriate language with items that are several sentences in length. Moderate complexity for this source also is characterized by moderate linguistic demands (i.e., generally, a few instances of some of the frequencies of all five linguistic demands; see above). High Complexity High complexity is defined as a combination of high reading demand and high linguistic demand. High reading load is characterized by grade appropriate language in prompts that are generally several sentences in length, with high linguistic demands (i.e., generally, instances of some of the frequencies of all five linguistic demands; see above). Complexity Sources that Were Considered and Not Included Here Amount of Scaffolding We interpreted this proposal from Achieve to address scaffolding within an item as, for example, when a complicated multistep item is broken into component steps and each step is designed to elicit a scorable examinee response. We determined that this type of scaffolding is (a) inherent in the design of an item, and (b) determined to some degree by other complexity sources, including the proficiency level target and Mathematical Process specified for each item. Number of Processing Steps Achieve proposed number of processing steps as a potential source of cognitive complexity. We believe that this potential source of cognitive complexity is addressed in Mathematical Process. For example, the definition for moderate complexity for Mathematical Process includes this statement: Items at this level involve more flexibility of thinking and choice among alternative methods of solution and/or response than do those in the low complexity category. The student is expected to decide what to do and how to do it, synthesizing concepts and processes from various areas of mathematics Composite Indices and Weights and Uses of the Indices We now propose how to weight, combine, and use each source of complexity in a composite index of cognitive complexity. A proposal for a different approach to creating final cognitive complexity indices, using empirical estimated decision trees, described in the document 4. Responses to change requests 08-31-12.pdf. Cognitive Complexity: Mathematics Page 8 Because weights like these are based on hypotheses and policy considerations, PARCC should consider their own hypotheses and policy considerations as they consider our recommendations. Cognitive Complexity Indices and Weights Mathematical Content and Practices are prominent in the Common Core State Standards because they are fundamentally important to mathematics teaching and learning. We also believe that they are the two primary sources of cognitive complexity in mathematics assessment items and tasks and drivers of item difficulty. The other sources of cognitive complexity are important, and relevant primarily in assessment rather than teaching. With those considerations in mind, we propose the following cognitive complexity indices and weights for each index in an overall cognitive complexity index. Proposed Weight in the Overall Index Rationale for the Weight Content Complexity 30% Practices Complexity 40% While content and practices are connected, factors listed under Practices are less dictated by the language of the Standards themselves, yet can lead to a wide range in the overall complexity of an item or task. Processing Complexity 30% Processing Complexity plays a significant role in item and task complexity, as indicated by empirical evidence. Overall Complexity 100% -- Proposed Index Note. Processing Complexity combines the sources Stimulus Material, Response Mode, and Processing Demands (all equally weighted). Rationales for the Proposed Weights Because this is an index of mathematical cognitive complexity, we propose that Content and Practices should contribute the most information. Item and task cognitive complexity that is influenced by processing demands inherent in test items and tasks together play a significant role in the overall index and, together, are approximately equal in importance to content or practice separately. These proposals would result in one overall measure of cognitive complexity, with five contributing individual sources. The overall index would be generated using one of the two options described in the Overview of this document. Each type I mathematics task (i.e., item) would be tagged with the overall index. Similarly, each prompt (i.e., item) in type II and III mathematics tasks would be tagged with the overall index. In addition, each type II and III task would be tagged with the overall index to aid forms assembly. Cognitive Complexity: Mathematics Page 9 Uses of the Indices We will use the Overall Index for cognitive complexity during item review processes. When questions arise about the complexity of an item or the appropriateness of an item because of its complexity, we will review the intermediate indices and individual sources measures. During the forms assembly process, PARCC and its vendors can use the Overall Index to select items, evaluate individual test forms, and to achieve parallelism of cognitive complexity across test forms, subject to the fundamental requirement of preserving the focus, coherence, and rigor of the Common Core State Standards through assessment. References Days, H.C., Wheatley, G. H., & Kulm, G. (1979). Problem structure, cognitive level, and problem-solving performance. Journal for Research in Mathematics Education, 10, 135-146. Ferrara, S., Svetina, D., Skucha, S., & Murphy, A. (2011). Test design with performance standards and achievement growth in mind. Educational Measurement: Issues and Practice, 30 (4), 3-15. Jerman, M., & Rees, R. (1972). Predicting the relative difficulty of verbal arithmetic problems. Educational Studies in Mathematics, 13, 269–287. Shaftel, J., Belton-Kocher, E., Glasnapp, D., & Poggio, J. (2006). The impact of language characteristics in mathematics items on the performance of English language learners and students with disabilities. Educational Assessment, 11(2), 105–126. Cognitive Complexity: Mathematics Page 10
© Copyright 2024 Paperzz