Don’t Guess . . .
Know When (and How)
to Assess
William F. McComas, Ph.D.
Director of Science Education Programs
Fellow, Center for Excellence in Teaching
Rossier School of Education
University of Southern California
Los Angeles, CA 90089-0031
[email protected]
What is
Assessment?
Assessment comes from the Latin
meaning to “sit with” a learner and
verify that the responses really
mean what they seem to mean
In most assessment plans such
assurance is lacking
Much assessment is suspect in
terms of validity and reliability
What Can We Assess?
Knowledge
Attitudes
Skills
Assessment Plans Must:
Be free of bias
Reflect what is (or should have been)
taught
Provide information to enhance
instruction (teaching or curriculum)
Reveal misconceptions
Adapted from Champagne, Lovitts and Calinger (1990).
Assessment in the Service of Instruction. Washington, AAAS.
Assessment should:
Be a learning experience for both students
and teachers
Involve the evaluation of reasonable
standards or benchmarks known in advance
to the learners
Be based on a well-designed table of
specifications

The table of specifications lists the learning goals
associated with the number, kind, and nature of
the individual assessment items
Timing is Everything
Diagnostic Evaluation (pretest)

Used for gauging prior conceptions on which
to base instructional strategies
Formative Evaluation (in process)

Useful for midcourse corrections
Summative Evaluation (posttest)

A way of seeing how well you did as a teacher
Sorry, Wrong Number!
Norm-referenced; measures of
performance against that of others

Scores are typically reported in percentiles

“Jane was in the 95th percentile on the SAT”
Criterion-referenced; absolute measures


Scores are typically reported in percents
“Bob got 82% of the questions correct”
Problems occur when instructors confuse
the two types of assessment goals
Major Assessment Issues
High Stakes Test; a single assessment
the result of which (positive or negative)
will have a large impact on the future of
the examinee

Certification, licensure, entrance exams, etc.
Low Stakes Test; a single assessment
the result of which does not have a large
impact on the future of the examinee

Most teacher-made tests
Major Issues in
Assessment Design
Reliability; a measure of the
consistency of the results of a given
instrument (mathematically determined)
Validity; a measure of the degree to
which the instrument measures what it
is designed to measure (qualitatively
determined)
What is Validity?
Validity refers to the accuracy of a
measure
A measurement is valid when it measures
what it is designed to measure and
performs the functions that it purports to
perform.
Does an indicator accurately measure the
variable that it is intended to measure?
What is Reliability?
Reliability refers to the dependability and
consistency of the results provided by an
instrument or technique.
A measurement is reliable when it produces
the same result repeatedly with the same
examinees or those from the same sample.
Does an indicator repeatedly report the
same measure that it is intended to
measure?
Forms of Assessment
Traditional:
typical forms
of style and
substance
Enhanced: new
forms of style,
substance and
goals
Traditional Assessment
Goals; assignment of
grades, student progress
reporting and fault finding
Target; learners
Timing; summative
Methods; objective
exams at the recall level
of the knowledge domain
Enhanced Assessment
Goals; assignment of
grades, student and
program assessment
Target; learners, instructors
and the curriculum
Timing; formative and
summative
Methods; objective exams
and expanded methods
targeting KSA at all levels
Enhanced Assessment Includes
More data points (not just summative
assessment)
More domains (not just knowledge)
but attitudes, skills, creativity, etc.,
Higher levels of all domains (such as
synthesis rather than memorization)
More techniques such as portfolio,
authentic assessment and empirically
derived exams
Why Expand Assessment?
The nature and focus of assessment tell
students (and teachers) what is important
Enhanced assessment encourages students
to achieve in nontraditional realms (ex;
creativity) and in nontraditional ways
Expanded assessment results provides the
data necessary to make more thoughtful
decisions about the curriculum, instruction,
and student progress
Multi-Domain Assessment
Knowing and Understanding
Exploring and Discovering (process skills
-- particularly in math and science)
Using and Applying Knowledge
Imagining and Creating
Feeling and Valuing (attitudes)
Understanding the nature of the discipline
(such as science)
Domain: Knowledge &
Understanding
Facts
Information
Concepts
Laws (principles)
Explanations and theories
Processes inherent to the discipline
Discipline-Based Skills and
Processes
Primary Science Process Skills
Observing
Using Space/Time Relationships
Classifying (Grouping and Ordering)
Using Numbers (Quantifying)
Measuring
Communicating
Inferring
Predicting
Discipline-Based Skills and
Processes
Integrated Science Process Skills
Controlling and Identifying
Variables
Interpreting Data
Formulating Hypotheses
Defining Operationally
Experimenting
Domain: Using & Applying
Seeing learned concepts in everyday life
Applying learned concepts to everyday life
Evaluating reports of scientific
developments
Making personal decisions based on
legitimate knowledge
Becoming involved in science-related
pursuits
Taking actions based on what has been
learned
Domain: Imagining & Creating
Visualizing by producing mental images
Combining objects and ideas in unusual
or useful new ways
Producing alternate uses for objects or
ideas
Pretending
Converging, Diverging and Synthesizing
Domain: Attitudes
Values and Feelings
Developing positive attitudes toward
science in general and school
science
Exploring and discussing both sides
of an issue
Expressing personal feelings in a
constructive fashion
Making decisions based on values
Domain: Understanding the
Nature of the Discipline
Appraising the strengths and limitations of
the discipline and its methods
Evaluating and applying appropriate
knowledge-production modes
Using criteria to place boundaries on the
discipline (what is in and what is out)
Understanding the social aspects of work
in the field
Bloom’s Taxonomy of
the Cognitive Domain
Knowledge (Recall of data)
Comprehension (Understanding)
Application (Using data)
Analysis (Separating ideas into their parts)
Synthesis (Building something out of parts)
Evaluation (Making judgments)
Technique:
Authentic Assessment
These are modes of assessment in which
examinees are asked to perform in ways
highly related to “real” situations (or
situations in which the learning originally
occurred)
“Real” situations are those that exist in
study, work, or life itself
The best authentic assessments are both
faithful (life-like) and comprehensive (wideranging)
Authentic Assessment
Involves:
Worthy problems or questions of importance
Faithful representations of real situations
Options, constraints, and access to resources
that are legitimate and not arbitrary
Problems that require a combination of
knowledge, judgment and creativity
Judgment standards that are clearly stated in
advance
Adapted from Wiggins (Nov / 93) Phi Delta Kappan
Technique: EmpiricallyDerived Exams
1 Determine the standards or bench-
marks for achievement
2 Create an open-ended assessment tool
3 Collect data and collapse similar
responses into typical response items
4 Now, use the typical responses on a
multiple choice type examination
Defining science is difficult because
science is complex and does many
things. But, MAINLY science is:
A the study of fields like biology, physics, etc.
B a body of knowledge (laws and theories) which
explain the world around us
C exploring the unknown and making discoveries
D carrying out experiments to solve problems
E inventing or designing things like computers
F an organization of people (scientists) who have
ideas for discovering new knowledge
G None of these choices fits my basic viewpoint
Technique: Performance
Assessment
Practical examinations,
Performances,
Exhibits
The task must be legitimate and
contextualized, not artificial and remote


in physical science - students measure the
temperature of a liquid vs.
in ecology - students determine the chemicals in a
sludge sample
Technique: Portfolio
Assessment
“. . . a systematic and organized
collection of evidence used by the
teacher and student to monitor
growth of the student’s knowledge,
skills, and attitudes in a specific
subject area.”
Vavrus, L. (August, 1990). Put portfolios to the test.
Instructor. pp. 48-53.
What Can Portfolios
Contain?
Artifacts; tests, lab reports, photographs,
meaningful journal entries
Reproductions; documents about typical
events not usually captured (videos of
presentations, photos of projects, etc.)
Attestations; something produced by
someone else (i.e. notes from the teacher)
Productions; documents prepared for the
portfolio (goal statement or reflection)
The Portfolio Process
Collect materials for the portfolio
Select materials to be in the portfolio
Reflect on why those materials were
selected
Project make a presentation based on
the portfolio contents
Respect the contents and presentation
Show the Effect of the portfolio
Issues in Enhanced
Assessment
How do we evaluate the results of
the new schemes?
How do we report the results?
What do we do with the results in
terms of advisement and promotion?
What are the solutions to questions
of time and expense?
Typical Small Toolbox of
Assessment Methods
Midterms and Finals
Multiple Choice Items
True False Items
Summative Assessment
Philosophy
Larger Toolbox of
Assessment Methods
Summative
Graded
• End of a Unit
• Feedback to students
•
True/False
Multiple Choice
Term Papers
Labs / Practical exams
Formative
• Not graded
•
•
•
•
Perhaps on-line
Feedback to all concerned
Immediate feedback
Allow for real time adjustments
Minute Paper
Empty Outlines
Approximate Analogies
Etc.
Don’t Guess . . .
Know you Know How
to Assess
William F. McComas, Ph.D.
Director of Science Education Programs
Fellow, Center for Excellence in Teaching
Rossier School of Education
University of Southern California
Los Angeles, CA 90089-0031
[email protected]