Method Template for Describing Alignment of State and NAEP Assessments
Part 1 – Alignment Method Overview
State and NAEP Assessment Alignment Method Guide
_______________________________________________________________
Purpose
This guide is designed to provide an overview and guidance for using a
structured method for assessing the alignment of your state’s assessment with
the NAEP (National Assessment of Educational Progress) assessment.
_______________________________________________________________
Use
The alignment method and associated materials described in this guide have
been developed for use as a job aid (step-by-step guidance) and contains:
 Part 1: Alignment Method Overview…describes the background, roles
and responsibilities, and the method and required documents
 Part 2: Administrator Guide…describes initial activities, completion of
the focus group tasks, completion of the state’s Department of Education
(DE) tasks, and the consolidation and reporting of results
 Part 3: Facilitator Guide…describes the role and specific steps required to
complete each task for the focus groups (separate guides for Mathematics
and Reading)
 Part 4: Appendixes A, B, C, D, E, F, and G contain NAEP content
expectations, released items, and frameworks for reading and
mathematics, as well as additional reference materials
 Part 5: Report Shell…contains a graph creation tool, a final report
template, and instructions for using them
_______________________________________________________________
Audience
This guide has been designed for use by education professionals within a
state’s DE office.
_______________________________________________________________
Additional
information
The NAEP content expectations and released items provided in this guide
were obtained from the NAEP website. Additional information regarding the
NCLB (No Child Left Behind) legislation may also be useful to the reader.
Websites:
(NAEP) http://nces.ed.gov/nationsreportcard
(NCLB) www.ed.gov/nclb/landing.jhtml?src=pb
_______________________________________________________________
Contents
Part 1: Alignment Method Overview contains the following information:
Section
Topics
Page
1.1
Background
2
1.2
Roles and Responsibilities
3
1.3
Method Description
5
_______________________________________________________________
Introduction
1
Method Template for Describing Alignment of State and NAEP Assessments
Part 1 – Alignment Method Overview
1.1 Background
_______________________________________________________________
Introduction
The No Child Left Behind (NCLB) Act of 2001 requires that all states design
and implement assessment systems to gauge student proficiency. Beginning
in the 2005-06 school year, NCLB requires that states annually test students
in Grades 3-8 in reading and mathematics. There are consequences for
schools that do not make progress toward proficiency goals by the year 2014,
with each state defining achievement and proficiency in its own way.
NCLB also requires that all states participate in the Grades 4 and 8 reading
and mathematics portions of the National Assessment of Educational Progress
(NAEP), with an implicit understanding that NAEP results can provide a
check on the progress reported by the states. The manner in which NAEP will
be used as a comparison for state-level results is unclear. However, since all
states must measure student proficiency levels and all must also participate in
NAEP, such comparisons are inevitable.
Important Note: The term “standard” in education is often used to describe,
specifically, what students should know, and be able to do, within content
areas. NAEP does not use the term “standards” to indicate what students
should know and do; however, in order to develop the assessment, they had to
identify similar levels which this alignment guide’s developers call “content
expectations.” Therefore, throughout this guide, we will use state “standards”
and NAEP “content expectations” as comparable terms.
_______________________________________________________________
Goal
The method described in this packet can be used by a state’s Department of
Education (DE) to make some judgments about the strength of the alignment
between the state and NAEP assessments. The state will undertake a series of
tasks, some to be performed in a focus group format (math or reading) and
others by state officials. When completed, these tasks will give DE officials a
more complete picture of similarities and differences between their state’s
assessment and the NAEP assessment.
_______________________________________________________________
Method
introduction
The assessment alignment process involves the completion of a series of tasks
(e.g., matching and comparing standards and content expectations, analyzing
test items, reviewing test scoring characteristics) in order to better understand
differences between state and NAEP assessments. Some tasks are completed
using teacher focus groups and others involve one or more state-level DE
personnel. The results of the tasks conducted by the focus groups and DE
staff are then documented and published in a customary fashion as directed by
each state.
_______________________________________________________________
Section 1.1 – Background
2
Method Template for Describing Alignment of State and NAEP Assessments
Part 1 – Alignment Method Overview
1.2 Roles and Responsibilities
_______________________________________________________________
Introduction
This method requires three key roles in successfully conducting an alignment
of state and NAEP assessments:
 administrator
 facilitator
 reviewer
_______________________________________________________________
Administrator
The administrator role should be filled by a state DE employee who is
knowledgeable about state and NAEP testing systems. The administrator will
oversee all facets of the alignment effort; responsibilities could be divided
between two or more people. If that is done, a formal coordination and
communication structure should be in place to ensure accountability.
Responsibilities include the following:
 Reading and understanding all materials provided in this guide
 Obtaining required materials such as state standards, released test
items, test blueprints
 Staffing focus groups (facilitators and reviewers)
 Focus group logistics (e.g., scheduling, meeting rooms, travel
accommodations, supplies)
 Scheduling DE staff and facilitating DE workgroup tasks
 Preparing for focus groups by sending “read-ahead” materials
 Remaining on site during focus groups to answer technical questions
about state and/or NAEP as they arise
 Rotating between focus groups to observe and help document the
process
 Publishing summaries or reports
_______________________________________________________________
Continued on next page
Section 1.2 – Roles and Responsibilities
3
Method Template for Describing Alignment of State and NAEP Assessments
Part 1 – Alignment Method Overview
1.2 Roles and Responsibilities, continued
Facilitator
The facilitator (one per focus group) role can be filled either by a state DE
staff member or a teacher. Facilitators should have some knowledge about the
state testing system and NAEP; however their primary responsibility is to
keep the focus group on task and document results. Previous experience as a
facilitator or group leader would be valuable.
Responsibilities include the following:
 Obtaining materials needed for focus group tasks from administrator
 Preparing for the focus group by reading and knowing the “readahead” materials and facilitator manual
 Leading discussions and focus group tasks
 Documenting the process and writing up preliminary results
_______________________________________________________________
Reviewer
There should be approximately five reviewers per focus group; one being a
state-level DE curriculum specialist, K-12. The other reviewers should be
current or recently retired teachers, some with experience at Grade 8 and
some at Grade 4. Teachers with experience on item- or standard-writing
committees for the state assessment would be especially useful.
Responsibilities include the following:
 Preparation for the focus group session by studying the “read-ahead”
material
 Active participation in focus group tasks
_______________________________________________________________
Section 1.2 – Roles and Responsibilities
4
Method Template for Describing Alignment of State and NAEP Assessments
Part 1 – Alignment Method Overview
1.3 Method Description
_______________________________________________________________
Alignment
method tasks
The tasks of the state and NAEP assessment alignment method are:
1. Comparing and matching state standards and NAEP content
expectations
2. Placing state and NAEP released test items on NAEP matrix (for
math, content strand by cognitive complexity level; for reading
context by aspect of reading)
3. Sorting state and NAEP items into a cognitive taxonomy (using state’s
taxonomy or using an existing taxonomy)
4. Comparing item format (types of questions, language, scaffolding)
5. Identifying reading passage (reading only) similarities or differences
6. Matching state operational test items to NAEP content expectations
(using the outcome product from Task 1)
7. Identifying differences between state and NAEP test administration
8. Determining scoring differences between the state and NAEP
assessments
_______________________________________________________________
Process for
Tasks
The tasks are conducted in two segments (in order):
 Tasks 1-5 (focus group tasks): The use of two content-specific focus
groups, one math and one reading; each group will work on both the 4th
and 8th grade levels. The recommended composition of each group is four
teachers, two each from a subject’s 4th and 8th grade level, plus one statelevel curriculum specialist.
 Tasks 6-8 (DE tasks): The use of appropriate DE staff (individual or small
group) based on the task being completed.
Once all tasks have been completed, the results of the alignment are
documented and reported as determined by individual state’s need. The length
of time required for this activity would vary greatly, depending on the type of
report, graphic complexity, and staffing availability. A report shell in Word®
has been provided in a separate folder (Report Shell), in addition to an
Excel® Spreadsheet to create graphs, should the administrator find them
helpful to use.
_______________________________________________________________
Continued on next page
Section 1.3 – Method Description
5
Method Template for Describing Alignment of State and NAEP Assessments
Part 1 – Alignment Method Overview
1.3 Method Description, continued
______________________________________________________________
Required
documents
The method uses two sources of materials: state materials (obtained from the
state’s DE) and NAEP materials (provided in Appendixes A through F).
State materials needed:
 Copies of reading and math assessed standards at Grades 4 and 8. It is
recommended that there be copies available of previous grade standards
as well (Grades 1-3 and 5-7).
 Copies of the most recent released state test form for reading and math,
Grades 4 and 8.
 Copy of the test blueprint that show the test’s structure. Blueprints
typically contain information on how many items per test, how these
items are divided among subjects (reading or math) and among domains
or strands (for example, number sense or reading for literary experience)
within the subjects.
 Copy of the system (taxonomy) your state uses to classify item
complexity. If your state does not have a published taxonomy, you may
use another taxonomy.
 Your state’s item documentation for operational test items. This
document should include information such as an item ID number, what
forms and grade level the item appears on, and the state standard to
which the item is linked.
NAEP materials provided:
 Copies of NAEP reading and math content expectations (Appendixes A
and B, respectively) to be used as “read-ahead” and focus group material.
 Copies of NAEP released reading and math items (Appendixes C and D,
respectively).
 Copies of NAEP reading and math frameworks, to be used both as “readahead” material for focus group participants and as reference during
focus group (Appendices E and F). If you are using this guide after Fall
2006, you should check the following websites to ensure the documents
are current:
 http://www.nagb.org/pubs/m_framework_05/toc.html
 http://www.nagb.org/pubs/r_framework_05/toc.html
_______________________________________________________________
Continued on next page
Section 1.3 – Method Description
6
Method Template for Describing Alignment of State and NAEP Assessments
Part 1 – Alignment Method Overview
1.3 Method Description, continued
______________________________________________________________
Alignment
method task
step description
Although specific guidance for each step is found in Part 3, Facilitator Guide
for Tasks 1-5 and Part 2 Administrator Guide for Tasks 6-8, the following
table provides a summary of the tasks.
Task
1. Match state
standards and NAEP
Description

content expectations
(Focus group task)


2. Match state and
NAEP test items
(Focus group task)




3. Sort items in
taxonomy (Focus
group task)
4. Compare test items
(Focus group task)






Reviewers will compare state standards and NAEP content
expectations; determining the degree of match between individual
standards: exact, partial, unique state, and unique NAEP
Results will be documented
For complete instructions, refer to Section 3.2.1 (Reading or Math)
Reviewers will sort NAEP released items onto the NAEP matrix
Next, they will sort state released items onto the same matrix,
attempting to match items by content strand and complexity level
(math) or reading context and aspect
Results will be documented
For complete instructions, refer to Section 3.2.2 (Reading or Math)
Using their state’s taxonomy, reviewers will sort state and NAEP
released items into taxonomic groupings
Results will be documented
For complete instructions, refer to Section 3.2.3 (Reading or Math)
Reviewers will examine state and NAEP released items for
similarities and differences in several categories:
 Type of questions (multiple choice, constructed response)
 Language used (specific versus general, use of verbs, etc.)
 Graphic elements (use of bold text, graphs, photos or drawings,
etc.)
Results will be documented
For complete instructions, refer to Section 3.2.4 (Reading or Math)
5. Compare reading
 Reviewers will compare state and NAEP released reading passages
passages (Focus group for similarities and differences in several categories:
task - reading only)
 Length of passage
 Reading level
 Bias
 Results will be documented
 For complete instructions, refer to Section 3.2.5 (Reading)
Table continued on next page
Section 1.3 – Method Description
7
Method Template for Describing Alignment of State and NAEP Assessments
Part 1 – Alignment Method Overview
1.3 Method Description, continued
Table continued from previous page
Task
6. Operational Items
(DE task)
Description
 DE personnel will link state operational test item documentation to
the groups of standards/content expectations produced in Task 1.
This task involves:
 Using operational item documentation to count the number of
times each standard is used as a primary, secondary, or tertiary
standard
 Using the worksheets from Task 1 that link state standards to
NAEP content expectations, enter the NAEP content
expectation and degree of match
 Results will be documented; graphs can be produced, such as the
State Standard bar graph sample on page 14, Administrator Guide
 For complete instructions, refer to Section 2.3.1 (Administrator
Guide)
7. Test Administration  DE personnel will review and document the similarities and
(DE task)
differences in the test administration between the state and NAEP
tests
 Use the guiding questions starting on page 19, Administrator Guide
 Results will be documented
 For complete instructions, refer to Section 2.3.2 (Administrator
Guide)
8. Score Computation
(DE task)
 DE personnel will review and document the similarities and
differences in score computation between the state and NAEP tests
 Use the guiding questions starting on page 27, Administrator Guide
 Results will be documented
 For complete instructions, refer to Section 2.3.3 (Administrator
Guide)
______________________________________________________________
Additional
Information
Specific information about tasks, materials, and duties of key personnel can
be found in Parts 2 and 3.
______________________________________________________________
Section 1.3 – Method Description
8