This video is for UNC Charlotte faculty
developing a response form as part of the
content validity protocol. The response form is
what expert reviewers will give feedback on
about your rubrics.
A separate video of directions for reviewers is
available on our COED website. Please do not
share this current video you are now watching
with reviewers.
Completing the initial review of rubrics
Selecting the panel of experts
Creating the materials for reviewers
Response Form
Assessment Packet
Collecting the information
Submitting the Data to COED Assessment
Rubric review has been ongoing
Rubrics must meet minimal CAEP guidelines
A checklist for this is on the COED Assessment
website under the “Content Protocol” link
Recommended date to have rubric changes
made is February 15, 2016.
Send your rubrics to Laura Hart for a quick
review before you proceed to the next step
Identify a panel of experts and credentials for
their selection.
Should include a mixture of IHE Faculty (i.e.,
content experts) and B12 school or community
practitioners (lay experts).
Minimal credentials for each expert should be
established by consensus from program faculty;
credentials should bear up to reasonable external
scrutiny
Invite them to participate, let them know they will
be receiving additional information in the near
future.
At least 3 content experts from the
program/department in the College of
Education at UNC Charlotte;
At least 1 external content expert from outside
the program/department. This person could be
from UNC Charlotte or from another IHE, as
long as the requisite content expertise is
established; and
At least 3 practitioner experts from the field.
TOTAL NUMBER OF EXPERTS: At least seven (7)
For each internally-development assessment/rubric,
there should be an accompanying response form that
panel members are asked to use to rate items that
appear on the rubric.
Multiple rubrics can be reviewed by the same panel of
reviewers, provided they have credentials.
Can be hard copy, electronic, survey monkey/survey
share … your preference
Example …
How Representative the Item is of Key Construct
How Important the Item is in measuring Key
Construct
How Clear the Item is
Rate each item on scale of 1-4
Representativeness of item in measuring the
overarching construct
1 = item is not representative
2 = item needs major revisions to be
representative
3 = item needs minor revisions to be
representative
4 = item is representative
Importance of item in measuring the overarching construct
1 = item is not necessary to measure the construct
2 = item provides some information but is not essential to
measure the construct
3 = item is useful not but essential to measure the
construct
4 = item is essential to measure the construct
Clarity of item
1 = item is not clear
2 = item needs some major revisions to
be clear
3 = item needs some minor revisions to
be clear
4 = item is clear
Key Construct
Provide response form
or link to it in
materials
Reviewer watches
directions video
Rate each item  how
well it measures key
constructs
Open-ended items:
Are there enough
items to measure the
construct? Too many?
Other feedback
Item 1
Item 2
Item 3
1.
2.
3.
4.
Hard copies or electronic versions
Letter of purpose (may be in the body of email)
A copy of the assessment instructions provided to
candidates.
A copy of the rubric used to evaluate the assessment.
The response form aligned with the assessment/rubric for
the panel member to rate each item.
Send materials to participants
Set a reasonable deadline
Once response data has been collected, submit
the complied results to the COED Assessment
Office.
Please do not submit individual results; please
compile the results into a summary chart or
document.
Copies of all forms and/or an excel file of
submitted scores (if collected electronically)
should be submitted in the designated file on the
S: drive.
This file is accessible by program directors (if you
need access, please contact Ashley Flatley in the
COED Assessment Office). Content Validity Results
are due by May 15, 2016.
There is a specific format – to see this and the file
path, go to the COED Assessment website.
Once Content Validity Results have been submitted, the
COED Assessment Office will generate a Content Validity
Index (CVI).
CVI = The number of experts who rated the item as 3 or 4 ÷
The number of total experts
A CVI score of .80 or higher will be considered acceptable.
Final changes will be made based on the CVI
results from the expert panel of review.
If we’ve done our work well, hopefully these are
small changes (or no changes) but some
adjustments will probably need to be made.
Final result = VALID RUBRIC
(interrater reliability becomes next step)
COED Website