Form C
ASSESSMENT REPORT
FOR
Psychology
(Instructional Degree Program)
Spring 2010 – Fall 2010
(Assessment Period Covered)
B.A., B.S.
(Degree Level)
11-15-10
(Date Submitted)
Intended Educational (Student) Outcome:
NOTE: There should be one form C for each intended outcome listed on form B. Intended outcome should
be restated in the box immediately below and the intended outcome number entered in the blank spaces.
Outcome 3: Successful students will understand and apply the fundamental theoretical perspectives of
psychology as a science
First Means of Assessment for Outcome Identified Above:
____a. Means of Program Assessment & Criteria for Success:
Means of Assessment
To facilitate a more objective analysis, the broad, over-arching outcome described above was
dissembled into six discrete, measureable sub-outcomes. These sub-outcomes had not yet been
established prior to the assessment process, so they were carefully developed at the outset of the
endeavor. In crafting these sub-outcomes, the assessment committee focused primarily on broad themes,
not discrete findings or principles. Moreover, our approach eschewed a conventional taxonomy that
categorizes knowledge according to distinct disciplines (e.g., Cognitive, Developmental, Social, etc.).
Rather, we aimed to create sub-outcomes that provide a flexible and enduring lens through which
students can interpret, categorize, assess and apply all psychological findings and principles.
Furthermore, the emphasis on a broad framework will help us reinforce the conceptual similarities and
interdependencies among the specific disciplines in a manner that is not availed by an examination of
distinct, separate fields. Ultimately, the sub-outcomes were selected to reflect both expert opinion (e.g.,
suggested outcomes from the American Psychological Association) and the essential tenets that define
the science of psychology.
Sub-outcomes
1) Psychology is a science. Successful students will understand that the theories and principles in the field
are proposed and tested using the scientific method, and conclusions are drawn based on empiricallyderived evidence.
2) Subjectivism vs. Objectivism. Successful students will understand that our conception of the world is
not merely an objective representation of incoming stimuli and information. Rather, our conception –
and perceptions – are shaped by a host of factors that reside within the individual, including prior
knowledge, present affect, and ordinary mental processes
3) External influences. Successful students will understand that an individual’s thoughts and behaviors
can be powerfully influenced by external factors, including other people and facets of the environment.
To properly predict and understand an individual’s thoughts and actions, these external influences must
be taken into account.
4) Mind-body interactions. Successful students will understand that the “mind” and body are not mutually
exclusive – most importantly, mental processes can exert a considerable influence on physiological states
and outcomes.
5) Heredity vs. environment. Successful students will understand that an individual’s traits and
development are shaped by a combination of hereditary (genetic) and environmental factors, often in a
synergistic fashion.
6) Need for consistency. Successful students will understand that individuals are compelled to maintain
consistency among their feelings, beliefs, and behavior. When one or more elements are inconsistent,
individuals will adapt their thoughts and/or actions to restore balance.
The assessment committee evaluated the targeted outcome using student artifacts based on two types
of assessments: multiple choice exam questions and essay exam questions. The artifacts were collected
from three lower division and five upper division courses: Introductory Psychology (PSY 101), Lifespan
Human Development (PSY 201), and Introduction to Research Methods (PSY 240) at the lower level, and
Health Psychology (PSY 470),Perception (PSY 405), Psychology of Aging (PSY 442), Social Psychology
(PSY 460), and Human Sexuality (PSY 466) at the upper level. Long form research and term papers were
not selected for assessment because it was believed that the specificity of such papers would be less
amenable to the examination of a broad, programmatic learning outcome than is afforded by a
constellation of questions taken from exams. To properly assess the breadth of the chosen outcome –
which targets theoretical perspectives that define the entire field of psychology – would have required an
array of long-form papers. The shorter but still substantive nature of the chosen exam questions allowed
us to assess a wider range of topics in a more manageable time frame.
Altogether, the assessment involved 22 multiple choice/fill-in-the-blank exam questions and nine
essay questions. For each essay question and discussion assignment, the committee randomly sampled
10 student artifacts from the corresponding class (yielding a total of 90 essay artifacts). For the multiple
choice questions, which are much less onerous to score and quantify, we simply looked at what
percentage of the entire class answered correctly. In sum, 499 multiple choice questions were scored for
the analysis.
Three independent raters assessed the quality of the essay artifacts using a detailed rubric based on
the six subcomponents described above. Student performance was assessed using the following fourpoint scale:
1 = Unsatisfactory – Student work represents an invalid or predominantly incomplete understanding of the
material (overall, this rating corresponds most closely to a ‘D’ or ‘F’ level performance).
2 = Satisfactory – Student work represents a largely valid understanding of the material, but at least one
aspect of their contribution is lacking in a noteworthy respect. The student’s treatment of the material is not
particularly thorough, and the quality of writing and critical thinking expressed in the student’s response is
rudimentary (overall, this rating corresponds most closely to a ‘C’ level performance).
3 = Proficient – Student work represents a valid understanding of the material, with no obvious
shortcomings. The student’s treatment of the material is fairly thorough, and the quality of writing and
critical thinking expressed by the student is good (overall, this rating corresponds most closely to a ‘B’ level
performance).
4 = Excellent – Student work represents a valid understanding of the material, with no obvious
shortcomings. The student’s treatment of the material is very thorough, and the quality of writing and
critical thinking expressed by the student is exemplary (overall, this rating corresponds most closely to an ‘A’
level performance).
Form C
Criteria for Success
For each individual essay artifact, success was operationalized as a score of 3 or higher (that is, a
demonstration of proficiency or better in a student’s understanding of a given sub-outcome). The
multiple choice/fill in the blank artifacts were defined as successful when at least 75% of the class
answered the question correctly. For the entire sample of artifacts, success was defined as a score of 3 or
higher for at least 75% of the sample. We also examined the extent to which the sample achieved a
satisfactory performance (a score of 2 or higher) and report these findings in this document.
____a. Summary of Assessment Data Collected:
Quantitative findings
Essay Questions
The committee analyzed the raw data set to determine the extent to which the three independent
raters agreed in their assessments of the student artifacts. Cronbach’s alphas, a statistical tool to
establish how well different raters rate the same item, did not fall below 0.76 for all items except one
essay question for Subjectivism. These analyses generally revealed a good degree of inter-rater
reliability, suggesting that the evaluation rubric permitted a relatively objective assessment of student
performance (consistently poor interpreter reliability would have indicated a much greater role of
subjectivity in our ratings, and a potential failure to properly assess how well students are performing).
The assessment committee discussed sporadic instances of low reliability for particular artifacts. Where
possible, differences based on a misunderstanding of the rubric were resolved (i.e., a few rater
disagreements were resolved by simply clarifying the intended meaning of the rubric); those that remain
stem from simple disagreements regarding the quality of an artifact.
Overall, the data suggest that student performance was, on average, satisfactory but not consistently
proficient. Across all essay/discussion artifacts, exactly 70% of the sample achieved a satisfactory score
or higher. In contrast, only 31% of the artifacts reflected a proficient level of competency. The
performance varied substantially by sub-outcome. Relatively strong scores were observed for Heredity
vs. Environment (at least 90% of the sample achieved a satisfactory rating and at least 80% were
proficient), passable scores for Psychology is a Science (75% satisfactory; 30% proficient), Mind-Body
Interactions (80% satisfactory; 30% proficient), and External Influences (70% satisfactory; 30%
proficient), and sub-par scores for Subjectivism vs. Objectivism (60% satisfactory; 10% proficient) and
Need for Consistency (60% satisfactory; 30% proficient).
A closer examination of the poorest performing domains – Need for Consistency and Subjectivism vs.
Objectivism – revealed two key findings. First, students performed poorly on one of the two Subjectivism
essay questions in large part because they relied more on anecdotal accounts and lay theories rather than
bona fide psychological principles. This particular question dealt with the interpretation of illness
symptoms, a topic that affects everyone and for which most people have cultivated personal theories or
adopted normative beliefs (e.g., eating chicken soup will diminish cold symptoms). In these and likeminded domains, some students express a tendency to rely on anecdotal information over principles
established by the scientific method.
Second, in regard to the sub-outcome of Consistency, students actually performed quite well on one of
the two essay questions that comprised the assessment. They faltered – badly – with an essay question
that required them to integrate two separate concepts in a novel fashion. The students’ answers
suggested that they understood each concept in isolation, but they were unable to bridge the ideas in a
manner permitting a satisfactory answer. The relatively inferior performance on this question indicates
that students may have placed too much emphasis on basic memorization and too little emphasis on the
critical thinking needed to solve novel problems.
Quantitative findings
Multiple Choice Questions
The committee linked each sub-outcome to two to four multiple choice questions. As noted under the
means of assessment, the committee examined the responses for the entire class for each multiple choice
question. All of the multiple choice questions involved a single objectively correct answer, so the
assessment committee quantified student performance simply by calculating the percentage of questions
answered correctly for each sub-outcome.
Overall, student performance for this mode of assessment was roughly on par with what we observed
on the essay questions. Correct answers were observed for roughly 65% of the artifacts analyzed (317
correct answers across 499 artifacts). When individual sub-outcomes were examined separately, student
performance on the multiple choice questions generally mirrored what was observed on the essay
questions. Students performed particularly well in the domains of Mind-Body Interactions (88% correct)
and Psychology is a Science (77%), performed modestly in External Influences (65%) and Need for
Consistency (60%), and struggled in the areas of Subjectivism vs. Objectivism (51%) and Heredity vs.
Environment (43%). The main point of divergence between essay and multiple choice questions occurred
for Heredity vs. Environment. The most likely explanation for the disparity is simply that the multiple
choice questions in this domain were significantly more challenging than the lone essay question selected
for the same sub-outcome. An alternative explanation is that the essay question was rated somewhat
leniently in this case, but given the high rate of agreement among the three independent raters, this is a
less viable explanation.
Quantitative findings
Overall
A comprehensive listing of student performance by assessment type (essay vs. multiple choice) and
sub-outcome is shown in the table below. Taken together, the data suggest that students enrolled in
psychology courses at NSC generally achieve a satisfactory understanding of the material, but a relatively
small proportion truly excel. This is hardly surprising given the open access nature of the institution, but
improving this performance will remain a point of concern as we move forward.
Two principle shortcomings emerged from the analysis, one major, pervasive issue and one minor
concern that afflicted a small proportion of students.
Major Area of Concern
Inadequate use of psychological principles and terminology
In several instances across all of the sub-outcomes, students who may have possessed a proficient
understanding of the material achieved only a satisfactory score because they failed to rely on
appropriate terminology. These students addressed the requirements of the assessment and provided
cogent answers, but their responses did not adequately convey a mastery of discrete psychological
concepts. Naturally, one objective of the program is to not merely improve student understanding of the
world around us, but to equip them with a knowledge of concepts and principles that provide a clear,
consistent framework for this understanding.
Minor Area of Concern
Failure to properly address specific facets of an assessment
A small but noteworthy contingent of students exhibited a peculiar – and unnerving – habit of
answering only a fraction of a multi-part question. This issue surfaced even among students who
Form C
demonstrated a high degree of competency (i.e., those who provided an exemplary response to part of an
essay question but completely neglected to answer a subsequent part). In the committee’s experience,
this issue is not endemic to the present assessment, but rather is characteristic of student performance in
a variety of domains. The problem typically is linked to a handful of students in a given class (or less), but
it nevertheless warrants attention.
Sub-outcome
1) Science
2) Subjectivism
3) External
4) Mind-body
5) Heredity
6) Consistency
Overall
Essay & Discussion Questions
Satisfactory or
Proficient or
higher
higher
75%
30%
60%
10%
70%
30%
80%
30%
90%
80%
60%
30%
70%
31%
Multiple Choice Questions
Correct responses
77%
51%
65%
88%
43%
60%
65%
___a. Use of Results to Improve Instructional Program:
The assessment committee has formulated several multi-faceted recommendations for how to
improve the program. One of these recommendations directly addresses performance shortcomings that
emerged in reference to the learning outcome assessed during this cycle, and two recommendations
address general strategies for improving the psychology program as a whole.
Specific Recommendation
1) Encourage students to incorporate specific psychological concepts and principles
As noted above, a major area of concern is the students’ failure to rely on specific, empirically-based
psychological findings and principles. The problem appears to manifest in two ways, and requires a
different solution for each. First, the issue is particularly evident in domains where people harbor
deeply-entrenched lay theories about how the world works (e.g., child development). As we know, these
lay theories are often flawed, especially when compared to the understanding we derive through the
scientific method, yet some students persist in subscribing to them. In this instance, we recommend that
faculty seek better and more insistent ways to illustrate the hazards of anecdotal evidence and the
benefits of scientific reasoning. For example, faculty could demonstrate the inherent shortcomings of
aphorisms. For every saying (e.g., “Absence makes the heart grow fonder”) there is an equal and opposite
saying (e.g., “Out of sight, out of mind.”). Rigorous scientific assessment is the most viable means of
reaching a useful, informative answer. In a related sense, faculty will be encouraged – through hiring
interviews, formal evaluations, and departmental materials (e.g., handbooks) – to work harder to show
students how the scientific methods can be applied to everyday events and decisions in an effective
manner.
Second, even among students who apparently understand and adopt psychological principles, there
was a tendency to omit the appropriate terminology. Consequently, faculty will be encouraged to revise
their assignment and exam instructions to better encourage the use of proper terms (and reward
students accordingly).
General Recommendations
1) Tie Learning Outcome to More Comprehensive Assessments
One of the key challenges posed by program assessment is the task of evaluating learning outcomes
conveyed by an entire baccalaureate degree through an analysis of specific assignments in particular
courses. No single course – much less a single assignment – can be expected to address entire
programmatic learning outcomes in their entirety. Nevertheless, the program and our students would be
well-served if major assessments in courses (key performances) more explicitly addressed specific
learning outcomes. Though the present comments are not based on a comprehensive analysis, an
examination of a substantive cross-section of assignments suggests that instructors are not indicating
which learning outcomes are being addressed by a particular assignment.
Nevada State College psychology students complete a substantive research paper in virtually every
class. These assignments typically require them to incorporate empirical evidence and discrete
principles from the discipline. However, the papers generally have not asked students to incorporate the
theoretical perspectives analyzed in the current assessment cycle. These broad outcomes (and suboutcomes) would not need to be the focus of a paper, but an explicit request to address one or more of
them would make students more cognizant of a widely applicable theoretical perspective. It would help
ensure that students do not lose sight of the forest for the trees, as it were. Alternatively, if a paper were
not amenable to the task of having students explicitly address a learning outcome/sub-outcome – or
simply ran contrary to the instructor’s intentions – the instructor could nevertheless indicate how a
particular assignment and principles it covers are tied to program-level learning outcomes. For example,
the instructions for a health psychology research paper on stress and immune function could remind
students of how these concepts reflect the over-arching theoretical perspective that deals with mindbody interactions. Ultimately, the effort on both counts is to increase the salience of our learning
outcomes (and sub-outcomes) and further ensure that students can use them as lens through which to
better understand and analyze information. One obvious means to meet this end is by modifying the
language in syllabi and assignment guidelines to better reflect the targeted learning outcome(s).
2) Create Outcome Map
The psychology faculty and corresponding assessment committee have emphasized the development
of meaningful, measurable learning outcomes for the psychology program. In assessing these outcomes,
we have examined the extent to which specific courses, typically between 2 and 5 in number, have
addressed these outcomes (as indicated by student mastery of relevant concepts). As we continue to
examine the integrity of specific outcomes, a next logical step is to determine whether the entire
spectrum of courses required by the major provides adequate coverage of the key outcomes – both in
terms of the present outcome being assessed, but also in regards to the programmatic objectives as a
whole.
The primary method by which to address this coverage is threefold. First, we would determine which
courses in the major should address particular outcomes and related sub-outcomes. Next, we would
examine our entire distribution of courses to create an outcome map that illustrates which outcomes
(and sub-outcomes) currently are being addressed by each course. Finally, we would try to resolve any
inadequacies or oversights – namely, learning outcomes that are not sufficiently addressed by the
program as a whole. Ultimately, the objective will be to establish a learning outcome map that indicates
which outcomes will likely be covered by particular courses and ensures that students will benefit from
“outcome saturation” – a deep and comprehensive treatment of learning outcomes across the program in
its entirety. Furthermore, instructors can improve student awareness of these outcomes by using syllabi
and assignment guidelines to indicate which outcomes are being addressed.
Form C
Moving forward, one key consideration will be to balance the desire to establish outcome saturation
against our belief that faculty should have ownership of their courses. The department – and generally
the entire School of Liberal Arts & Sciences – subscribes to a philosophy that empowers talented faculty
to take ownership of their classes. As such, instructors will not be forced to adopt particular learning
outcomes in specific classes, but rather will be shown how the adoption works to the benefit of our
students. If this approach fails to achieve the desired outcome saturation, the social sciences chair and
psychology faculty will discuss alternative pathways.