How well did the assessment task
do what we wanted it to do?
Janina Drazek
Manager — Assessment & Comparability, QCAR
Queensland Studies Authority
Central role of the teacher
Skilled diagnosticians, i.e. teachers
diagnose and assess against
declared standards.
Diagnosticians require reliable
instruments designed to produce valid
information.
ASSESSMENT
is the teacher’s instrument.
What is assessment?
Assessment is the purposeful, systematic and ongoing
collection of information as EVIDENCE for use in making
judgments about student learning.
» (Education Queensland 2001, Years 1-10 Curriculum framework for
Education Queensland schools, Department of Education, Brisbane, p.13)
To develop valid and reliable tasks
three reflective steps are required:
1. Evaluation
2. Modification
3. Trialling
When and how do I EVALUATE?
ALL the time. In particular,
1. once a task has been created do the task yourself.
2. once you trial the task with learners—give them the
possibility of commenting on the task.
Using all the information you have gained modify the task
or reject entirely.
Q. BUT I did all that, now I want to
know how do I evaluate whether
the task worked?
A. Evaluate by applying exactly the
same principles used for the
development of quality assessments.
Sound assessment design
Selecting a proper method to:
• suit the purpose and the target
• elicit the right performance or product
• provide enough evidence to support the
decision
• avoids sources of bias.
Evaluating
Ask the students
• Did they enjoy the task?
• Was it easy/difficult?
• Did they understand what to do?
• How well did they think they responded to the task?
• Ask focused questions about how they did the various
sections of the task.
Evaluating
Look at the evidence (student work)
• Mark the responses according to the criteria and
standards matrix (schema, rubric etc)
• Note the range of standards in the responses
(consider and analyse)
• Do the responses reflect your indicative
response? If not, why?
Evaluating
Review the task against the principles of
quality assessment
• Credibility–valid and reliable
assess what it purports to assess
• Intellectual rigor
focus on higher order thinking
• Authenticity
have purpose
• User friendliness
feasible, appropriate and engaging
fair and equitable
Credibility Are the criteria explicit?
Does the task
generate
sound
evidence
about student
performance?
Are the standards clearly stated?
Is it clearly aligned with the
targeted intentions?
Are the conditions of assessment
clear?
How fair and equitable to all
students?
Does it provide opportunity for
students to demonstrate what they
know and can do?
Evaluating
Criteria
• Did they reflect what is valued in the assessment?
• How were they organised?
• Were they organised under: Knowing and understanding;
Inquiring; Responding; Reflecting (or some other
constructs)?
• Look at the number of criteria – too many or too few?
• Are they sufficiently different from, and independent of,
each other?
• Did the students understand what was expected?
Evaluating
Standards Descriptors
• cover a range of performances on a particular
criterion
• are sufficient in number to enable differentiation
of performances
• can be seen in student work
• are written in positive terms
• are written in language that is suitable for
students
Intellectual draws in sufficient depth & breadth
upon the targeted knowledge,
quality
Is the task
intellectually
challenging?
concepts and skills
engages students in a range of
thinking skills
encourages students to
demonstrate critical analysis
has clear cognitive expectations
Authenticity
Does the task
address a
realistic and
complex
problem?
has an appropriate real life
context
for all students
is engaging and motivating
has genuine and valued purpose
has a context that is appropriate
for
the cognitive demands
User
has an accessible & appropriate
friendliness format
Does the task
clearly
communicate
to students
what is
needed for
producing
their best
performance?
has layout, cues, visual design,
format and choice of words that
clarify what is expected
has examples and resources that
are
helpful to students
provides a complete set of
guidelines
(including models) that allow
students to reflect on, rehearse and
review their responses
Alignment of Assessment &
Curriculum
1.
Intent
(What do we want students to learn?)
2.
Enactment
(What are the appropriate learning
experiences?)
3.
Evidence
(What are the contexts in which students
demonstrate their learning?)
4.
Judgment
(What is being valued?)
5.
Moderation
(Is there consistency in the way we apply
standards?)
Effective assessment design
• depends upon a clear and complete explication
of the claims one wishes to make on the basis of
assessment, the evidence needed to support
those claims, the student behaviour that will
comprise that evidence, and the tasks needed to
elicit those behaviours. That chain of reasoning,
once established and documented becomes part
of the validity argument in support of the
assessment (Matters, 2006:14).
QCAR contacts
Website: www.qsa.qld.edu.au/qcar
Email: [email protected]