Designing High Quality, Affordable
Assessment Systems
Edward Roeber
Michigan State University
National Research Council Board on Testing and Assessment
April 6-7, 2010
1
Overview

Balanced Assessment Systems



Summative Assessment Designs







Horizontally Balanced
Vertically Balanced
Purpose of this Study
Typical Current Assessment Program
High Quality Assessment Program
Development and Administration Costs
Cost Reduction Strategies
Results
Recommendations
2
Vertically-Balanced System

Summative Component


Interim Benchmark Component


Instructionally-relevant, short-cycle summative assessments
also using a broad array of assessment types
Formative Assessment Component



Broad array of types of assessment
Adequate professional development
High-quality pre-service education
All three components present, and equally important to
the teacher and the student
3
Vertically-Balanced System

Portions of this system are out of balance:







Interim benchmark assessments not thoughtfully created,
acquired, or used
Educators don’t understand how to use interim assessments
Only conventional types of assessment used
Lack of understanding about what formative assessment is and
is not (e.g., it is not an item bank)
Lack of opportunities for educators to learn how to use formative
assessment strategies as they teach
Educators don’t learn about classroom and formative
assessment in pre-service programs
State summative assessments overpower the interim and
formative assessments in the minds of educators (and students)
4
Horizontally-Balanced Systems






Multiple-choice items are the predominant form of
assessment used by states
Some states use constructed-response items
Fewer still use performance assessments
Emphasis is on speed of return of results, not quality
or usefulness of the information once it gets there
Schools feel compelled to “teach to the test” in some
not-so-good ways - some subjects not taught at all
and the rest are taught to the extent that they are on
the test
Only state-assessed skills are focused on in local
assessments
5
Horizontally-Balanced Systems

This system is out of balance in some important ways

More types of items should be used in the system
• Short- and extended-response items
• Performance events
• Performance tasks



This broader array of items could have a positive
impact on teaching and learning - could be
assessments worth teaching to
These state assessments could also serve as a model
for developing instructionally-sensitive interim
assessments
The prime question is can states can afford to use this
broader array of assessment?
6
Purpose of This Cost Study


Determine the cost of a typical state assessment program that
primarily uses multiple-choice items
Design a high quality summative assessment (HQA) as just
described




Note - HQA designed for Mathematics and Reading/Writing only. This excludes Science,
alternate assessments for students with disabilities and ELPA assessments
Determine the cost of such a HQA
Determine whether several potential strategies could reduce costs
of the HQA significantly
Present various options and costs for an interim assessment system
similar to the HQA
7
Current Typical Summative
Assessment Design
Summative Assessment Design
Summative Assessment
Mathematics
Item Counts
Short Constructed Extended Constructed Performance
Response
Response
Event
Multiple Choice
Performance Task
Current Typical
Assessment
Summative Assessment
English Language Arts
50
0
2
0
Item Counts
Short Constructed Extended Constructed Performance
Response
Response
Event
Multiple Choice
0
Performance Task
Current Typical
Assessment (Reading)
50
0
2
0
0
10
0
1
0
0
Current Typical
Assessment (Writing)*
8
Current Typical Interim
Assessment Design
Interim Assessment Design
Interim Assessment
Mathematics
Item Counts
Short Constructed Extended Constructed Performance
Response
Response
Event
Multiple Choice
Performance Task
Current Typical
Assessment
Interim Assessment
English Language Arts
40
0
0
0
Item Counts
Short Constructed Extended Constructed Performance
Response
Response
Event
Multiple Choice
0
Performance Task
Current Typical
Assessment
40
0
0
0
0
9
High Quality Summative
Assessment Design
Summative Assessment Design
Summative Assessment
Mathematics
Item Counts
Short Constructed Extended Constructed
Response
Response
Multiple Choice
High Quality
Assessment
Summative Assessment
English Language Arts
25
25
High Quality
Assessment (Writing)*
2
(1 in grade 3)
(0 in gr. 3, 1 in gr. 4)
Item Counts
Short Constructed Extended Constructed
Response
Response
Multiple Choice
High Quality
Assessment (Reading)
2
10
2
2
(1 in gr. 3 & 4)
(1 in gr. 3 & 4)
2
2
(1 in gr. 3 & 4)
(1 in gr. 3 & 4)
Performance
Event
Performance Task
2
2
(0 in gr. 3, 1 in gr. 4)
Performance
Event
Performance Task
2
1
2
0
10
High Quality Interim Assessment
Design
Interim Assessment Design
Interim Assessment
Mathematics
Item Counts
Short Constructed Extended Constructed
Response
Response
Multiple Choice
High Quality
Assessment
Interim Assessment
English Language Arts
Performance
Event
1
25
2
(0 in grade 3)
Item Counts
Short Constructed Extended Constructed
Response
Response
Multiple Choice
Performance Task
1
1
(0 in grade 3)
Performance
Event
Performance Task
1
1
High Quality
Assessment
25
2
1
11
Development and Administration
Costs

Typical Assessment Program - $20/Student


HQA Summative Assessment Program - $55/Student


Cost per student for the current typical assessment calculated using
the ASG cost model (includes Year 0 development costs)
Cost per student for the high quality assessment calculated assuming
a single state implementation and no cost reduction strategies
(includes Year 0 development costs)
Most states cannot afford a nearly tripling of their state
assessment costs, so the result will be a limit on innovative
assessment types, unless something occurs
12
Cost Reduction Strategies

Participation in a consortium




Uses of technology for online test delivery, distributed human
scoring of some of the open-ended items, and automated scoring
for certain constructed response items



Model includes state consortium sizes of 10, 20, and 30 states
Use of a state consortium reduces costs by an average of $15 per student
Consortium approach represents a significant decrease in assessment cost
Together, these innovations account for cost savings of about $3 to $4 per
student
Likely to account for more as efficiencies are developed in programming and
using technology for these purposes
Two approaches to the use of teacher-moderated scoring.
Teacher-moderated scoring can net both substantial cost
reductions as well as potential professional development benefits.
We used two different models for teacher-moderated scoring
13
Cost Reduction Strategies

Two different models for teacher-moderated scoring:


Professional development model - no additional teacher compensation
beyond that supported by the state or district for normal professional
development days
Stipend model - assume a $125/day stipend for teachers to score the
performance items.
•


Note: teachers were assumed to score all performance items in a distributed
scoring model
These strategies for using teachers as scorers reduce costs by an
additional $10 to $20 per pupil (depending on whether teachers are
engaged as part of professional development or are paid)
Adopting all cost reduction strategies while paying teachers a
$125/day stipend to score all performance tasks results in an
assessment cost of $21
14
Overall Cost Reduction Results
Diminishing Expenditures - per Student HQA Cost
$60.00
Cost per Student
$50.00
$40.00
$30.00
$20.00
$10.00
$HQA
50/50
Distributed
Scoring on
SCR/ECR
On-Line Testing Teacher Scoring Teacher Scoring HQA - All cost
w/Stipend
w/o Stipend
variables
applied
w/stipend
Single State
10 State
20 State
HQA - All cost
variables
applied w/o
stipend
Current MC
(Non-HQA Test)
30 State
15
Impact of Teacher Scoring Time
on Costs
10 State Consortia - PE & PT Scoring
Cost per Student - Impact of Increase in Scoring Time
$45.00
$41.21
$40.00
$36.64
Cost per Student
$35.00
$32.06
$27.48
$30.00
$22.91
$25.00
$18.33
$20.00
$15.00
$27.08
$24.83
$22.57
$20.32
$13.76
$18.06
$15.82
$10.00
$13.56
$5.00
$Base
1 min
2 min
3 min
4 min
5 min
6 min
Number of Minutes Added
PE Cost per Student
PT Cost per Student
16
Cost Study Conclusions

The development cost of a new HQA is relatively inexpensive
relative to the total cost of the assessment



A key factor in determining whether states can adopt and sustain new
improved assessments is ongoing administration costs
In order to reduce costs, states should participate in an assessment
consortium to share the overhead associated with development,
administration, and management of assessments
Larger consortia are somewhat more cost-effective


The majority of cost savings relative to the single state case are seen even in at
a 10-state consortium size
States should strongly consider being part of a large consortium where certain
costs can be shared across many states, such as for item development and
project management
17
Cost Study Conclusions

Implementing a HQA system with performance items is affordable,
with teacher scoring of performance items at a price comparable to
today’s assessments, when procured by a consortium of states



In order to implement and afford an HQA system that includes a variety of
performance items, it is essential to have teachers involved in the scoring
process
The cost impact of increases in the time to score performance items is very
significant
The use of online technology (i.e., online assessments) should be
encouraged


It has the potential to reduce assessment cost and improve quality
The procurement of PCs to improve the student- to-PC ratio should be
encouraged at all levels of the educational system
18
Recommendations

Developing and implementing an HQA will likely cost more than
most current state assessments. It can be affordable for states if
they look carefully at





the design of the summative assessment component
finding a balance in the number of CR items, PEs, and PTs used
consider various cost-reduction strategies
It is recommended that state consortia go about the process of
designing a new assessment in a thoughtful manner, then use a
comprehensive costing model to analyze and determine the price
in advance of any new assessment system they would like to
implement
State consortia interested in implementing a HQA should make
sure they can afford the ongoing administration costs of the
assessment before they embark on developing it
19
For More Information
Edward Roeber
Michigan State University
263 Erickson Hall
East Lansing, MI 48824
(517) 432-0427
[email protected]
For a copy of the complete paper for this study, go to the
Assessment Solutions Group website to download a copy:
www.assessmentgroup.org
20