Cost Estimate for the New
Kentucky Assessment System
Prepared for:
Ken Draut
Roger Ervin
Jennifer Stafford
Kentucky Department of Education
Report Prepared by Assessment Solutions Group
September 28, 2009
Contents






Report Overview
Methodology
2012 Assessment System Key Elements and
Assumptions
Program Costs and Key Cost Components
Summary, Conclusions and Recommendations
Appendices
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
2





Report Overview
 Purpose/Scope
 Uses
 Potential range of bids
 Nature of the Kentucky assessment
Methodology
New Assessment System Key Elements and
Assumptions
Program Costs and Key Cost Components
Summary, Conclusions and Recommendations
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
3
Report Overview/Scope


The purpose of this report is to provide the KDE assessment
staff with a realistic estimate of the cost for their new
assessment system.
The assessment components studied, analyzed, and modeled by
Assessment Solutions Group (ASG) included:




A combined CRT/NRT assessment in the domains of math, reading, science
and social studies
A writing assessment
A writing editing and mechanics exam
ASG did not examine costs for other assessment components,
including the ACT exams, Alternate Assessments, and English
Language Proficiency Exams. ASG also did not model any
interim or formative assessment components in this study.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
4
Report Overview
(cont.)


For each assessment studied, assessment elements were
identified at the individual cost component/work level and the
associated costs then modeled and calculated using ASG’s
proprietary cost modeling software.
This allows for an accurate costing of the proposed, actual
Kentucky assessment system for 2012.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
5
Uses of the Report

The primary use of this report is to provide the KDE with a realistic
estimate of the cost of the key components of its new 2012
assessment system that can be used for budgeting and planning
purposes.



Modeling the costs of the new system using the actual design of the new
assessment program will provide an accurate estimate of the costs KDE should
expect to see from an efficient, realistically pricing vendor when it bids the
program in 2010
ASG believes its costs are accurate to a level of +/- 8% to 10% although industry
spreads on bids tends to be far greater than this range
ASG’s model outputs also allow KDE to gain a deeper understanding
of the costs of the various components of each assessment and
therefore provides KDE the ability to make trade-offs in assessment
program design prior to competitive bidding in order to develop the
highest quality, cost-effective assessment for the budgeted dollars.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
6
Potential Range of
Actual Program Bids


As stated previously, while ASG believes its cost projections are
accurate to within 8%-10%, it is likely that KDE will see a much
larger spread from vendors actually bidding on the program.
This is due to several factors including:


Nature of the Kentucky program (combined CRT/NRT, 6 different
assessments) and potential bidders
Past history of NCLB bids



Vendor bidding strategies and cost structures
Vendor understanding of the RFP
ASG will address each of these factors briefly in the following
pages.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
7
Nature of the
Kentucky Assessment



The combined CRT/NRT assessments for reading, math, science,
and social studies may narrow the field of bidders (to NRT
owners) and reduce competition in bid pricing. Mitigating this
somewhat is that the NRT owners may be willing to either partner
with and/or license their content to other test companies.
Two of the three NRTs are 4 color tests which are more
expensive to produce than 2 color tests.
The three NRT vendors are all in somewhat different competitive
positions in the industry which can lead to different pricing
strategies

Information redacted
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
8
Nature of the
Kentucky Assessment
(cont.)

The new Kentucky assessment is relatively large with 6 content
areas to be tested, including 4 with a combined CRT/NRT.
Experience has shown that the larger the assessment, the more
moving parts and the greater potential range of prices.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
9
NCLB Bid History


Past bids on NCLB assessment programs have shown an
extremely wide disparity. ASG believes the large spreads on bid
pricing will decrease in the future and that prices, in general,
will rise as vendors seek to improve profitability in this segment.
ASG believes the disparity in NCLB bid pricing is due to several
factors including differing vendor pricing strategies in the early
years of NCLB and a lack of a clear understanding of program
requirements.

It can be difficult for States to write a good RFP that clearly outlines
program requirements comprehensively and in sufficient detail


Lack of RFP clarity results in vendors making assumptions as to how a program is to be
developed, performed, and/or administered.
The vendor tends to make assumptions based on the least cost alternative in order to arrive at
the most competitive bid
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
10
NCLB Bid History
(cont.)
When developing the RFP for the new Kentucky
assessment program, it will be critical to have a clearly
written proposal that calls for the vendor to submit the
metrics and specifications outlining their understanding
of the program requirements with their response.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
11


Report Overview
Methodology






ASG Cost Model Overview
Data Gathering and Input
Data Review and Analysis
New Assessment System Key Elements and
Assumptions
Program Costs and Key Cost Components
Summary, Conclusions and Recommendations
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
12
Methodology –
ASG Cost Model


The ASG Cost Model was used to develop the cost of the new
Kentucky assessment system components.
The ASG Cost Model is a variable input, metric-based calculation
output model.


Specific assessment program variables are input to the model and applied
against cost factors to derive assessment cost
Several hundred variables are contained in the model.


Numerous detailed input variables are assembled for each of the functions
and elements involved in constructing an assessment (test development,
production and manufacturing, logistics, editing and scoring, psychometrics,
reporting, program management, quality assurance, information
technology, etc.)
These variables are input into the software and matched to metric based
calculations/formulas and/or databases to convert the assessment program
variables into dollar costs
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
13
Methodology –
ASG Cost Model



(cont.)
The cost for the entire assessment is built from the ground up,
element by element, function by function.
Proprietary functions and calculations have been developed
based on ASG’s years of experience in the industry.
Model outputs have been tested against real state data.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
14
Methodology –
Data Gathering
The information required to develop the price for the new
Kentucky assessment system consisted of the following steps:
1. ASG staff read the new state law, SB1, and various documents from
the KDE summarizing the new state assessment
2. Several discussions with KDE management (Ken Draut, Roger Ervin,
Jennifer Stafford) allowed ASG to gain a deeper understanding of the
features of the new assessment system and details of the assessment
system components
3. An ASG data input sheet consisting of the key elements required to be
input to the ASG cost model was completed by KDE personnel
4. ASG held a meeting with KDE to review and understand all data input
sheet elements and state assumptions
5. ASG submitted and KDE responded to a series of questions and
clarifying assumptions after the initial data entry and data output
review was done by ASG
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
15
Methodology –
Data Input & Review
6. Initial, high level cost estimates were shared with KDE to insure both
sides were in the same ballpark on cost expectations
7. ASG input the revised programmatic data into its cost model and
generated various outputs and reports
8. The data was reviewed by Senior ASG consultants (Topol and Olson),
assumptions were questioned/revised and the data were rerun. Four
to five iterations of this process occurred before the data were
finalized.
9. Final revisions were made and final outputs (data tables and charts)
generated for inclusion in this report.
10. The cost estimate report and executive summary were written,
reviewed, and revised several times until they reached their current,
final state.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
16





Report Overview
Methodology
New Assessment System Key Elements and
Assumptions
Program Costs and Key Cost Components
Summary, Conclusions and Recommendations
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
17
New Kentucky 2012
Assessment System –
Elements & Assumptions

The new 2012 Kentucky assessment system consists of the
following elements:

A combined CRT/NRT in








Math
Reading
Science
Social Studies
grades
grades
grades
grades
A writing test
A writing editing & mechanics exam
3-8, 11
3-8, 10
4, 7, 11
5, 8, 11
grades 4, 6, 8, 10, 12
grades 4, 8, 12
Student counts are roughly 50,000 per grade (see the appendix
for the exact student counts by grade and other assumptions).
In order to have a valid number of responses to measure
student performance on each of the 10 KY principles
(standards) per domain, an assessment (combined CRT/NRT) of
91 questions will be required (7-10 items per principle).
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
18
New Kentucky 2012
Assessment System –
Elements & Assumptions

The assessments will consist of both multiple choice and
constructed response questions.







(cont.)
Math, English, Science, and Social Studies assessments will each include 4
constructed response items (mix of short and extended response). The
writing exam will include 2 extended response items and the writing editing
and mechanics exam will include 1 constructed response item.
Scoring for all open ended questions will be done on site
Two forms of each assessment will be developed for year 1
(2012) and a new form developed each subsequent year.
There will be a 3% item release rate per year, per assessment.
Test books are assumed to be in 4 color to accommodate the
current color formats of NRTs (Stanford and TerraNova).
Testing will be done with paper and pencil.
All reports (except for the parent report) will be posted online.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
19
New Kentucky Assessment
System – Elements &
Assumptions (cont.)





Assessments will be given to students during the last two weeks
of the school year (May – early June).
Results will be mailed to parents and posted online in August.
Shipping will be to school districts using ground transportation.
One large standard setting meeting will occur, concurrently for
all content areas, in the first year of the new assessment.
Assessments will be given to students during the last two weeks
of the school year.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
20
New Kentucky Assessment
System – Elements &
Assumptions (cont.)

It is assumed the current vendor will conduct the
field test of the new items during the 2011
administration.




Either the new or current vendor can develop the items
It is assumed that half of the CRT items required can be
obtained from the existing item bank for use in 2011-2014
It also is assumed that the current vendor will have enough
test forms in which to place all the field test items for the
administration in the 2011 year
Roughly 10% of the student count, or 5,000, will be
sufficient to generate reliable and accurate statistics
on the field test items.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
21
ASG Pricing Philosophy
& Key Assumptions
ASG adopted a relatively conservative philosophy/
methodology to arrive at the price of the
Kentucky assessment system

First, ASG believes prices in the industry will be increasing

Information redacted
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
22
High Level Assumptions
and Cost Considerations
(cont.)

Second, competition for the new Kentucky program could be
somewhat limited due to the NRT component of the exam


While ASG does not expect this to result in a higher price to KDE, it could
have a limiting impact on the potential number of bidders (excluding the
partnering/licensing issues)
Third, the purpose of this exercise is to provide an estimate that
KDE can use for budgeting purposes so conservatism is key
Due to the above factors, ASG assumes vendors will price
new programs at profit margins between 20% - 40%,
after covering all overhead
For year 0 activities, ASG assumes a 50% margin ,which is
typical of “transition year” programs
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
23


Report Overview
Methodology







ASG Cost Model Overview
Data Gathering and Input
Key Assumptions
Data Review and Analysis
New Assessment System Key Elements and
Assumptions
Program Costs and Key Cost Components
Summary, Conclusions and Recommendations
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
Confidential
24
Program Costs and
Key Cost Components


ASG calculated the costs of the assessment program for years 0,
1, 2, and 3 (2011, 2012, 2013, 2014)
Activities in year 0 primarily include development and field
testing of the items required to create two new forms of the
assessment for 2012


Some psychometric work is also required
Year 0 costs represent the incremental costs (in addition to the
regular 2011 assessment costs) and are possibly subject to
greater variability as the exact details of the current contract
were not included in the scope for this report. Therefore, the
item development plan for 2011 is open to further discussion on
its design and implementation. However, costs for this work are
included in the report.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
25
Program Costs and
Key Cost Components




The first year of administration is 2012 (year 1) and consists of
all activities to conduct an operational administration of the new
assessment and for development of new items to create a new
form in 2013.
One large standard setting meeting will be held in year 1
(2012).
Thereafter, the program remains fairly constant in the following
two years (2013 and 2014).
ASG has assumed a 4% price escalator for years 2-4.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
26
Year 0 (2011) Costs
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
Year 0 Costs by
Function
Costs of activities related to Year 0 by functional area – includes
estimates with 50% and 30% margin
Function
Content Development
Composition/Pre Press
Printing
Open Ended Scoring
Psychometrics
Grand Total
September 28, 2009
Activity
Content Acquisition/Item Procurement
Art Procurement
Item Development
Forms Construction
Ancillary Development
Administrative
Subtotal
Page Composition
Translation (Braille)
Subtotal
Film Prep & Plate
Print & Binding
Paper
Custom/Ancillary
Subtotal
Scoring
Training/Training Prep/Mgmt
Travel
Subtotal
Labor
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
50% Margin
248,313
108,333
261,337
187,442
98,280
147,409
1,051,113
139,369
376,272
515,641
144,286
500,931
5,859
212,355
863,431
627,575
93,600
721,175
79,102
3,230,462
Assessment Solutions Group
www.assessmentgroup.org
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
30 % Margin
177,366
77,381
186,669
133,887
70,200
105,292
750,795
99,549
268,766
368,315
103,061
357,808
4,185
151,682
616,737
448,268
66,857
515,125
56,501
2,307,473
28
Year 0 Cost by
Functional Area
Cost by functional area for Year 0 (2011)
Note that ASG’s base case assumes year 0 work is priced at a 50% margin, which is
typical of “transition year” programs. Year 0 work also is shown at a 30% margin.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
29
Year 1 (2012) Costs
Years 2-3 (2013-2014) Costs
and Total Costs
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
30
Costs by Functional
Area & Cost per Student
Year 1 (2012) cost for each functional area and cost per student
Cost per Student Breakdown
Total Cost
1.5%
Cost per Student
2.4%
Content Development
Composition/Pre Press
Printing
Warehouse/Distribution
Freight Costs
Receiving/Scanning/Machine Scoring
Open Ended Scoring
Score Reporting
Quality Assurance
Psychometrics
Technology Support
Program Management
Total
$
$
$
$
$
$
$
$
$
$
$
$
$
1,651,325
378,957
1,759,796
287,404
1,013,851
863,741
4,696,106
295,374
144,690
184,152
493,564
766,309
12,535,268
September 28, 2009
$
$
$
$
$
$
$
$
$
$
$
$
$
3.82
0.88
4.07
0.66
2.34
2.00
10.85
0.68
0.33
0.43
1.14
1.77
28.96
1.2%
3.9%
6.1%
13.2%
3.0%
14.0%
37.5%
8.1%
2.3%
6.9%
Content Development
Composition/Pre Press
Printing
Warehouse/Distribution
Freight Costs
Receiving/Scanning/Machine Scoring
Open Ended Scoring
Score Reporting
Quality Assurance
Psychometrics
Technology Support
Program Management
Assessment Solutions Group
www.assessmentgroup.org
31
Total Program Costs
Total program costs for 2011 - 2014
2011
Program Costs
2012
2013
2014
Total
Base Cost
$
1,242,485 $
7,121,851 $
7,161,759 $
7,448,229 $
22,974,325
Overhead
$
372,746 $
1,957,646 $
1,982,198 $
2,061,486 $
6,374,076
Margin*
$
1,615,231 $
3,455,771 $
3,514,016 $
3,654,577 $
12,239,595
Total
$
3,230,462
$
12,535,268
$
12,657,974
$
13,164,292
$
41,587,996
*Margin rate is 50% for 2011 and 30% in subsequent years. The exception is travel and NRT licensing, which are set at 15%
Total Program Cost % Split: 2011 - 2014
29.4%
55.2%
15.3%
Base Cost
September 28, 2009
Overhead
Margin*
Assessment Solutions Group
www.assessmentgroup.org
32
Additional Assumptions and
Considerations – NRT Cost Impact


The impact on costs of a combined CRT/NRT is not
expected to result in a higher price for the state vs. use of
a CRT only assessment.
While there are only 3 NRT vendors, ASG has talked to two of the
three and has determined:


Information redacted
Riverside’s NRT (ITBS) is in black and white (2 color) and tends to
be priced below either Stanford or TerraNova which are both in 4
color.

Information redacted
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
33
Additional Assumptions and
Considerations – Test/Scoring
Window




Kentucky’s tight scoring window is not expected to
result in increased assessment costs, however any
significant testing delay in a number of schools could
pose some problems.
ASG calculates roughly a 3 week scoring window for the vendor
Such a scoring window is tight but ASG does not view it as
being out of the reasonable range of performance for a major
test vendor. Furthermore, the scoring window for the Kentucky
assessment will fall outside the busiest scoring period for most
contractors thereby easing the strain on them, provided other
states do not move their administration and scoring dates to
later in the school year.
ASG has run a case to estimate the cost impact of a compressed
scoring window and included it in the range of outcomes (see pp 52)
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
34
Additional Cost Data for
New Assessment Program






Costs by Functional Area
Cost per Student
Cost by Content Area
Cost by Grade
Cost by Content Area and Grade
Total Costs and Key Metrics
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
35
Kentucky Assessment
Program Cost Data
Costs for the Kentucky program by functional area for 2012 - 2014
Total Cost by Function
Content Development
Composition/Pre Press
Printing
Warehouse/Distribution
Receiving/Scanning/Machine Scoring
Open Ended Scoring
Score Reporting
Program Management
Quality Assurance
Psychometrics
Technology Support
Grand Total
September 28, 2009
$
$
2012
2013
2014
1,651,325 $
378,957
1,759,796
1,301,255
863,741
4,696,106
295,374
766,309
144,690
184,152
493,564
12,535,268 $
1,456,422 $
394,115
1,830,188
1,353,305
898,290
4,883,950
307,189
796,961
118,282
191,518
427,753
12,657,974 $
1,514,679
409,880
1,903,395
1,407,437
934,222
5,079,308
319,477
828,840
123,013
199,178
444,863
13,164,292
Assessment Solutions Group
www.assessmentgroup.org
Total
$
$
$
$
$
$
$
$
$
$
$
$
4,622,426
1,182,951
5,493,379
4,061,997
2,696,253
14,659,365
922,040
2,392,110
385,985
574,848
1,366,179
38,357,534
36
Cost per Student
Cost per student for the Kentucky assessment program for the
years 2012-2014
2012
2013
2014
Avg Cost
Total Cost per Student by Function
Content Development
Composition/Pre Press
Printing
Warehouse/Distribution
Receiving/Scanning/Machine Scoring
Open Ended Scoring
Score Reporting
Program Management
Quality Assurance
Psychometrics
Technology Support
Grand Total
September 28, 2009
$
$
$
$
$
$
$
$
$
$
$
$
3.82
0.88
4.07
3.01
2.00
10.85
0.68
1.77
0.33
0.43
1.14
28.96
$
$
$
$
$
$
$
$
$
$
$
$
3.37
0.91
4.23
3.13
2.08
11.28
0.71
1.84
0.27
0.44
0.99
29.25
$
$
$
$
$
$
$
$
$
$
$
$
Assessment Solutions Group
www.assessmentgroup.org
3.50
0.95
4.40
3.25
2.16
11.74
0.74
1.92
0.28
0.46
1.03
30.42
$
$
$
$
$
$
$
$
$
$
$
$
3.56
0.91
4.23
3.13
2.08
11.29
0.71
1.84
0.30
0.44
1.05
29.54
37
Cost by Content Area
2012 total cost by content area for the Kentucky assessment program
2012 - Cost by Content Area
$964,463
$3,612,433
$2,223,060
$1,249,855
$1,315,057
Reading
September 28, 2009
Math
$3,170,401
Science
Social Studies
Writing
Editing & Mech
Assessment Solutions Group
www.assessmentgroup.org
38
Cost by Content Area
Cost by content area for the years 2012 - 2014
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
39
Cost by Grade
Costs for the Kentucky assessment program by grade for 2012 - 2014.
Note: grades 4 and 8 are tested more often; grade 3 uses consumable test books
2012
Total Cost per Grade
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade 10
Grade 11
Grade 12
Grand Total
September 28, 2009
$
$
$
$
$
$
$
$
$
$
$
2013
1,290,123
2,085,552
1,379,098
1,388,489
1,363,475
2,183,805
956,381
1,202,516
685,829
12,535,268
$
$
$
$
$
$
$
$
$
$
$
1,311,013
2,103,288
1,388,449
1,405,356
1,372,070
2,205,645
971,182
1,205,976
694,995
12,657,974
2014
$
$
$
$
$
$
$
$
$
$
$
1,363,453
2,187,420
1,443,987
1,461,570
1,426,953
2,293,870
1,010,029
1,254,216
722,795
13,164,292
Assessment Solutions Group
www.assessmentgroup.org
Total
$
$
$
$
$
$
$
$
$
$
$
3,964,589
6,376,260
4,211,534
4,255,415
4,162,498
6,683,320
2,937,592
3,662,708
2,103,618
38,357,534
40
Cost by
Content Area & Grade
Costs for the Kentucky assessment program by content area & grade for 2012 – 2014
Grade 3
2012
Reading
Math
Science
Social Studies
Writing
Editing & Mech
Grand Total
2013
Reading
Math
Science
Social Studies
Writing
Editing & Mech
Grand Total
2014
Reading
Math
Science
Social Studies
Writing
Editing & Mech
Grand Total
$
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
679,265
610,858
$ 1,290,123
$
489,427
416,430
458,430
394,353
326,913
$ 2,085,552
$
495,275
453,523
430,300
$ 1,379,098
$
505,919
426,915
455,656
$ 1,388,489
$
474,068
442,961
446,445
$ 1,363,475
$
505,998
421,675
425,260
478,598
352,274
$ 2,183,805
$
$
691,078
619,934
1,311,013
$
493,646
417,730
461,409
401,909
328,594
2,103,288
$
499,815
456,393
432,241
1,388,449
$
510,884
428,720
465,751
1,405,356
$
477,717
445,365
448,988
1,372,070
$
510,924
423,227
426,956
489,567
354,970
2,205,645
$
718,721
644,732
1,363,453
$
513,392
434,439
479,865
417,985
341,738
2,187,420
$
519,808
474,649
449,530
1,443,987
$
531,320
445,869
484,381
1,461,570
$
496,825
463,180
466,948
1,426,953
$
531,361
440,156
444,035
509,150
369,169
2,293,870
September 28, 2009
Grade 10
-
$
$
-
$
-
$
Grade 11
Grade 12
462,480
493,901
956,381
$
398,039
410,182
394,295
$ 1,202,516
$
$
465,683
505,499
971,182
$
399,081
411,709
395,187
1,205,976
$
484,310
525,719
1,010,029
$
415,044
428,178
410,994
1,254,216
$
Assessment Solutions Group
www.assessmentgroup.org
Total
400,552
285,277
685,829
$ 3,612,433
3,170,401
1,315,057
1,249,855
2,223,060
964,463
$ 12,535,268
$
409,051
285,945
694,995
$ 3,649,747
3,190,451
1,322,107
1,254,384
2,271,776
969,509
12,657,974
$
425,413
297,382
722,795
$ 3,795,737
3,318,069
1,374,991
1,304,559
2,362,647
1,008,290
13,164,292
$
41
Total Costs and Key
Metrics by Year
Total program costs and key metrics by year for 2012 - 2014
Unit of Measure
Items Acquired
Permissions
Total Items Developed
Art Pieces
NA
NA
NCOunt
2012
Total Cost Total Units Cost per Unit
$
4,524
158 $
28.56
$
135,386
65 $ 2,089.29
$
325,697
158 $ 2,056.17
$
14,295
128 $
111.75
$
17,335
NA
NA
$
654,733
NA
NA
$
499,356
390,800 $
1.28
2013
Total Cost Total Units Cost per Unit
$
4,705
158 $
29.70
$
140,801
65 $ 2,172.86
$
245,897
158 $ 1,552.38
$
14,866
128 $
116.22
$
18,028
NA
NA
$
512,795
NA
NA
$
519,330
390,800 $
1.33
Functional Area
Content Development
Cost Item
Item Acquisition
Permission Costs
Item Development
Art Costs
Item Banking
Development Travel
NRT Licensing
Composition/Pre-Press
Total Composition Costs Pages Composed
$
378,957
6,700 $
Printing
Total Printing Costs
Bound Units
$ 1,759,796
3,203,741 $
Warehouse/Distribution
Total Warehouse Costs
Freight Out Cost
Scoring Freight In
Pounds Shipped
Pounds Shipped
Pounds Returned
$
$
$
287,404
427,331
586,520
Receiving/Scanning/Machine
Scoring
Receiving/Order Entry
Document Prep/Staging
Image Scanning
Document Editing
Documents
Documents
Sheets Scanned
N-Count
$
$
$
$
38,778
468,487
95,804
260,671
Open Ended Scoring
Total Cost
Scores
$ 4,696,106
6,511,586 $
0.72 $ 4,883,950
6,511,586 $
Score Reporting
Total Cost
Pages Printed
$
295,374
1,731,200 $
0.17 $
307,189
1,731,200 $
All Areas
Total Program Cost
N-Count
$ 12,535,268
432,800 $
28.96 $ 12,657,974
432,800 $
September 28, 2009
383,502 $
383,502 $
350,106 $
1,356,100
1,356,100
6,512,200
432,800
$
$
$
$
56.56 $
394,115
6,700 $
0.55 $ 1,830,188
3,203,741 $
0.75 $
1.11 $
1.68 $
298,900
444,424
609,981
0.03
0.35
0.01
0.60
40,329
487,227
99,636
271,098
$
$
$
$
Assessment Solutions Group
www.assessmentgroup.org
383,502 $
383,502 $
350,106 $
1,356,100
1,356,100
6,512,200
432,800
$
$
$
$
2014
Total Cost Total Units Cost per Unit
$
4,893
158 $
30.89
$
146,433
65 $ 2,259.77
$
255,732
158 $ 1,614.47
$
15,461
128 $
120.86
$
18,750
NA
NA
$
533,307
NA
NA
$
540,103
390,800 $
1.38
58.82 $
409,880
6,700 $
61.18
0.57 $ 1,903,395
3,203,741 $
0.59
383,502 $
383,502 $
350,106 $
0.81
1.21
1.81
0.78 $
1.16 $
1.74 $
310,856
462,201
634,380
0.03
0.36
0.02
0.63
41,942
506,716
103,622
281,942
$
$
$
$
1,356,100
1,356,100
6,512,200
432,800
$
$
$
$
0.03
0.37
0.02
0.65
0.75 $ 5,079,308
6,511,586 $
0.78
0.18 $
319,477
1,731,200 $
0.18
29.25 $ 13,164,292
432,800 $
30.42
42
Programmatic Cost
Variables Impact
ASG calculated the impact on total costs for various
changes in the following program components








A margin change of +/- 10%
Impact of using a 2 color vs. a 4 color format
Impact of item bank usage to populate the CRT items. The calculated
percentages include 100%, 50% (level assumed for costing), and 0%
Impact of halving the number of constructed response questions
Impact of having a 25% item release rate vs. a 3% rate (level for costing)
Impact of lower shipping rates
Impact of using distributed scoring for CR items
Impact of a 10% cost adjustment imposed due to the tight scoring window
Note that the items above are not ASG recommendations but are
presented to provide valuable programmatic cost information
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
43
Summary of Impact of
Options/Changes in Costs
Chart shows cost savings or increases related to various options/changes to the
program for 2012 - 2014
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
44
Impact of Vendor
Margin Change
Impact on annual and total costs of a 10% change in contractor margin
$15,296,943
$16,000,000
$14,708,599
$14,560,978
$15,000,000
$13,164,292
$14,000,000
$12,657,974
$12,535,268
$13,000,000
$12,000,000
$11,555,595
$11,111,149
$11,006,461
$11,000,000
$10,000,000
2012
2013
-10% Margin
September 28, 2009
Current Margin
2014
+10% Margin
Assessment Solutions Group
www.assessmentgroup.org
45
Impact of Using 2
Color vs. 4 Color
Impact of producing the test in 2 color vs. 4 color (base case). Note: this will require
psychometric approval and perhaps some equating studies for the NRT
component.
$16,000,000
$15,000,000
$13,164,292
$14,000,000
$12,657,974
$12,535,268
$13,000,000
$12,118,383
$12,713,389
$12,224,413
$12,000,000
$11,000,000
$10,000,000
2012
2013
2 Color Test Book
September 28, 2009
2014
4 Color Test Book
Assessment Solutions Group
www.assessmentgroup.org
46
Impact of Use of the Existing Item
Bank For CRT Items – Year 0
Chart reflects total cost in year zero based on usage of the existing item
bank to generate 100% , 50%, or 0% of the CRT items.
$4,289,286
$4,000,000
$3,750,000
$3,230,462
$3,500,000
$3,250,000
$2,763,203
$3,000,000
$2,750,000
$2,500,000
$2,250,000
$2,000,000
100% CRT from Item Bank
September 28, 2009
Current (50% CRT from Bank)
No CRT from Item Bank
Assessment Solutions Group
www.assessmentgroup.org
47
Impact of halving the number of
constructed response items on
total costs
Impact on total costs of halving the number of constructed response
items in Years 1, 2, and 3
$16,000,000
$15,000,000
$13,164,292
$14,000,000
$12,657,974
$12,535,268
$13,000,000
$12,000,000
$11,200,621
$10,720,581
$10,770,699
$11,000,000
$10,000,000
2012
2013
Half Constructed Response
September 28, 2009
2014
Full Constructed Response
Assessment Solutions Group
www.assessmentgroup.org
48
Impact of a 25% Item
Release Plan
Impact on total costs of a 25% item release rate vs. the current 3% rate
$16,000,000
$15,000,000
$13,358,518
$14,000,000
$12,535,268
$12,759,469
$12,657,974 $12,844,729
$13,164,292
$13,000,000
$12,000,000
$11,000,000
$10,000,000
2012
2013
3% Item Release
September 28, 2009
2014
25% Item Release
Assessment Solutions Group
www.assessmentgroup.org
49
Impact of Reduced
Shipping Rates
Impact of lower vendor shipping rates on costs vs. standard UPS rates
assumed in the model
$16,000,000
$15,000,000
$13,164,292
$14,000,000
$13,000,000
$12,893,738
$12,285,125
$12,535,268
$12,397,825
$12,657,974
$12,000,000
$11,000,000
$10,000,000
2012
2013
Discounted Freight
September 28, 2009
2014
Standard Freight
Assessment Solutions Group
www.assessmentgroup.org
50
Impact of Distributed
Scoring For CR Items
Impact of using “work from home” readers in distributed scoring
$16,000,000
$15,000,000
$13,164,292
$14,000,000
$12,657,974
$12,535,268
$13,000,000
$11,896,823
$12,473,750
$11,993,990
$12,000,000
$11,000,000
$10,000,000
2012
2013
Distributed Scoring
September 28, 2009
2014
Site Scoring
Assessment Solutions Group
www.assessmentgroup.org
51
Impact of Cost Adjustment
for Tight Scoring Window
Impact of 10% adjustment in applicable labor costs for processing/scoring
$16,000,000
$15,000,000
$13,657,310
$14,000,000
$12,991,090
$12,535,268
$13,132,029
$13,164,292
$12,657,974
$13,000,000
$12,000,000
$11,000,000
$10,000,000
2012
2013
Standard Rate
September 28, 2009
2014
10% Cost Adjustment
Assessment Solutions Group
www.assessmentgroup.org
52
Summary, Conclusions,
& Recommendations

ASG performed a special study analyzing the costs of the new
Kentucky assessment system






Year 0 (2011) activities – development and field testing
Years 1 – 3 (2012-2014) – full administration of new assessment
A vendor performing efficiently is expected to price the year 0
activities at $3.2M and the total program costs for years 20122014 at $12.5M, $12.7M, and $13.2M respectively
ASG performed a number of scenarios to consider what could
change the cost picture significantly
Consistent with other NCLB state programs, the highest costs
are in the item development, scoring, and printing areas
Metrics and benchmarks used in the analysis represent those of
an efficient contractor realistically bidding on the program
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
53
Summary, Conclusions
& Recommendations




ASG adopted a relatively conservative methodology to
arrive at the price of the Kentucky assessment system
using fully burdened costs and reasonable margin
assumptions.
ASG’s analysis finds the total cost and cost per student of
the new Kentucky assessment system appear to be in a
reasonable range for a high stakes, NCLB-type state
assessment.
Kentucky’s new assessment itself is more comprehensive
than many other state assessments that ASG has seen.
Combining this assessment with other testing elements
(ACT), as well as with interim or formative assessments,
should result in a well-balanced and comprehensive
assessment system.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
54
Summary, Conclusions
& Recommendations



The proposed item release rate and use of NRT
components require only one form per year which
significantly controls costs.
The impact on costs of a combined CRT/NRT is not
expected to result in a higher price for the state vs.
use of a CRT-only assessment.
ASG assumes that Kentucky’s tight scoring window
should not result in increased assessment costs,
however this depends on the vendor’s master
schedule and other factors. A significant testing
delay in a number of schools and late return of
materials could pose some problems.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
55
Summary, Conclusions
& Recommendations



ASG recommends that the KDE consider ways to
improve the cost-effectiveness and efficiency of the
new assessment system in the future.
When developing the RFP for the new Kentucky
assessment program it will be critical to have a
clearly written scope of work that calls for the
vendor to submit the metrics and specifications
outlining their understanding of program
requirements with their response.
As Kentucky’s PC-to-student ratio improves, ASG
recommends online testing be considered.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
56
Appendix





List of assumptions for new KY assessment program
Other uses of the ASG cost model
About ASG
Contact Information
References
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
57
List of Assumptions
Content Developm ent
Design/Com position/Printing
Warehouse/Distribution
Open Ended Scoring
Score Reporting
Psychom etrics
Program Managem ent/Travel/Other
No NRT item development w ill take place. Items w ill be part of NRT product from vendor
CRT Items for year 1 w ill be populated from both a field test and the existing Kentucky Item Bank
Item bank w ill provide 50% of CRT for year 1 test
2 forms w ill be developed for year 1.
NRT Licensing Fee costed at a rate of $1 per student
No alternate language versions of the assessment for ELL w ill be required
Only one form w ill be composed and printed each year
All test books are assumed to be 4 color, w ith grade 3 being a consumable test book
DFA/Coordinator manuals are assumed to be 2 color versus 1 color
Braille test books are a ratio of 3 pages to every one page of a standard test book
Large print test books are a ratio of 2 pages to every one page of a standard test book
All paper used for test books and answ er documents is 60# off w hite
Answ er Document includes Standard 2 page answ er documents for CRT plus additional pages to support constructed response items
Field test in 2011 to test items for 2012 administration w ill be supported by existing vendor
Exiting vendor can imbed field test items in existing assessment w ith only additional form cost incurred
Outbound materials w ill be shipped standard ground delivery
Inbound consumable test books and answ er documents w ill be returned 2nd day freight
Non consumable test books w ill be returned via standard ground shipping
Standard UPS shipping rates are used to calculate freight costs
Read behind/check score rate is set at 20%
For field test items, 10% of students responses w ill be scored
Site scoring w ill be used for all open ended scoring
Printed reports w ill be done in 2 color format
All reports w ill be shipped via 2nd day shipping
Parent report w ill be a 2 page report, w ith 2 copies printed
All other reports w ill be delivered electronically
Psychometrics labor is allocated to standard setting, item analysis and equating/scaling, w riting Technical Report and attending TAC meetings
Program w ill have full time dedicated program manager
Standard Government GSA rates are used for lodging and per diem travel expense
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
58
Number of Students
By Grade
Grade
Number of Students
03
50,000
04
50,000
05
49,000
06
49,000
07
49,500
08
49,500
09
51,000
10
49,300
11
44,500
12
42,000
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
59
Other Uses of the
ASG Cost Model

Once an RFP is let, the ASG model can be used to evaluate
actual bids to ensure vendors understand the program and are
submitting “apples to apples” responses




ASG cost sheets included in the proposal provide key metrics that allow KDE
to ensure all vendors understand the key elements of the program
The cost sheets and metrics also provide a firm basis to negotiate with
vendors on potential cost reductions
The additional level of data ASG provides also allows KDE to work with
vendors to make trade-offs in design elements without impacting program
quality
The model is also valuable in determining the correct amount to
pay for future program scope changes
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
60
About ASG
The Assessment Solutions Group (ASG) is a consulting organization with a
mission of assisting state departments of education in adding value
throughout assessment costing, procurement and management
functions. ASG senior consultants and technical advisors have more
than 100 years combined experience in the assessment industry and
expertise in all areas of the assessment function, making ASG unique in
the industry in being able to provide states with services in test
development, psychometrics, IT, production and manufacturing, quality
assurance, scoring operations, and logistics. ASG makes extensive use
of its proprietary costing model in providing services to its customers in
the areas of cost-effective and efficient assessment program design.
The company’s other product offerings include RFP preparation and
analysis, technical and cost proposal reviews, ongoing assessment
program evaluation, and program management services.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
61
John F. Olson, Ph.D.
John F. Olson serves as senior consultant and partner in the Assessment Solutions Group (ASG), which he
co-founded in 2008. Dr. Olson also is President of the consulting business he founded in 2006, Olson
Educational Measurement & Assessment Services, which provides technical assistance and support to
states, the U.S. Department of Education, and others. Previously, he served as Vice President for
Psychometrics and Research Services at Harcourt Assessment, Inc. From 1998 to 2003, Dr. Olson
was Director of Assessment for the Council of Chief State School Officers (CCSSO), where he worked
with state leaders on an assortment of educational assessment projects. Prior to that, he was Deputy
Director of the Center for Education Assessment at the American Institutes for Research where he
managed the Voluntary National Tests program. From 1985 to 1995, he held several increasingly
important positions at the Educational Testing Service (ETS), including Director of Operations, Director
of State Services, and Director of Technical Assistance for NAEP.
Dr. Olson has more than 25 years of experience providing professional services to state DOEs, the USED,
testing companies, researchers, and others on a variety of measurement and statistical issues for
international, national, state, and local assessment programs. In addition, he has an extensive
research and publications record. His expertise in the areas of large-scale assessment, test design
and development, testing program management and operations, psychometrics and analysis, NCLB,
NAEP, and quality assurance is of much value to customers. Dr. Olson holds a Ph.D. in educational
statistics and measurement from the University of Nebraska - Lincoln.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
62
Barry Topol
Barry Topol, CPA - Principal
Barry Topol is a strategic and operationally oriented Financial Executive with over 25 years
experience. He has worked in industry as a Chief Financial Officer for several major corporations
and small companies and as an independent consultant. He is a co-founder of ASG. Mr. Topol
has a broad business background with C-Level experience in educational publishing, high
technology, internet and other industries. His expertise includes activity based costing,
turnaround situations, detailed financial and operational analysis, strategic planning and
leadership and team-building. Topol became interested in improving the educational
assessment process during the period from 2005-2008 when he was the CFO for a major
educational publishing company. He has developed numerous cost models for educational
assessments and has experience creating new pricing frameworks for assessment services. He is
a skilled negotiator and communicator, working effectively with management, vendors and
other constituents on behalf of the represented entity.
Mr. Topol has a BA in economics from UCLA where he graduated Summa Cum Laude and was
elected to Phi Beta Kappa. He earned an MBA from UCLA’s Anderson School of Business where
he won the Charles Offer Foundation Fellowship and was an Edward W. Carter fellow.
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
63
Contact Information


This report was written by Barry Topol and John Olson, ASG Cofounders and Senior Consultants
For more information, contact
 Barry Topol, [email protected]
 John Olson, [email protected]
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
64
References



Kentucky Senate Bill 1 (SB1, July 8, 2009)
Kentucky Department of Education Office of Assessment and
Accountability information and updates on SB1 (KDE website)
Assessment Data Worksheet Kentucky Assumptions and
Questions (memo transmitted Sept. 2, 2009)
September 28, 2009
Assessment Solutions Group
www.assessmentgroup.org
65