Outcomes Assessment: Reporting the Results for Academic Year 2013-20141
Name of Program: Masters in Public Administration (MPA)
Program Director: Dr. Christine Martell
Person Completing Report: Dr. Christine Martell and Vanessa Fenley
Date: June 30, 2014
This report evaluates the University of Colorado Denver, School of Public Affairs’ MPA program. The
MPA program is a professional degree that is designed to produce: 1) generalist executives in the
public and non-profit sectors, such as city managers, agency CEOs, department heads and mid-level
managers; 2) specialists in such fields as local government and urban affairs, environmental law and
policy, non-profit management, domestic violence, emergency management, and policy analysis and
program evaluation; 3) program managers in such areas as human services, environmental protection,
natural resource conservation, disaster management, domestic violence, local government, health, and
administration of cultural and artistic programs; 4) elected officials in state and local government, and;
5) corporate positions in public advocacy or affairs or in regulated areas.
The program has developed four learning objectives which reflect the knowledge and skills our
faculty believe students should obtain to earn their MPA. Students should be able to:
1) Demonstrate a broad base of knowledge concerning public or nonprofit management, policy,
and research;
2) Use effective critical thinking and analytical skills to solve problems in public and nonprofit
settings in a manner that will improve the public good;
3) Communicate effectively in writing; and
4) Communicate effectively in speaking.
Additionally, there are five identified competencies2 that we have evaluated in this report. Students
should be able to:
1) Lead and manage in public governance;
2) Participate in the policy process;
3) Analyze, synthesize, think critically, solve problems, and make decisions;
4) Articulate and apply a public service perspective; and
5) Communicate and interact productively with a diverse and changing workforce and citizenry.
Finally, the program has two outcomes that reflect our goals:
1) Graduates will work in professional management or policy positions in public or nonprofit
settings.
2) Graduates will make use of the knowledge and skills they have gained to perform effectively in
their positions.
This MPA outcomes assessment is designed to measure the above learning objectives, competencies,
and outcomes.
1
This report contains information for the academic year 2013/2014, which includes the summer and fall of 2013, and
spring of 2014.
2
These competencies are required by NASPAA, our accrediting body.
1
The following measurement instruments were used to assess our objectives, competencies, and
outcomes:
1. Rubric for Capstone Seminar: The rubric is designed to assess students’ actual performance.
Students complete the MPA program with a capstone class, PUAD 5361 Capstone Seminar. In
this course, they complete an applied research project designed to demonstrate the knowledge
and skills they have gained in the program. Students select a client-based project and are
assessed by three readers: the primary instructor, another SPA faculty member, and a client or
external reader. The rubrics rate students as below proficient, proficient, and above proficient in
each of the four objectives. Readers are also provided an opportunity to submit additional
comments on each student’s project.
2. Capstone Inventory of Competencies: The five competencies are further delineated into 19 subcompetencies that provide operational definitions. The capstone inventory records which of the
19 sub-competencies a student’s capstone project addresses. As not all projects capture each
sub-competency, this measure gives us an overall view of what competencies students express
in their capstone.
3. MPA Student Exit Survey: After students complete the capstone course, they are asked to
complete a student exit survey. The survey is given to students as part their graduation packet,
but filling out the survey is optional. The student exit survey asks students to assess the MPA
program’s effectiveness and contribution to improving students’ knowledge and oral, written,
and communication skills, meeting students’ educational needs, improving student’s abilities in
their current work positions, and improving students’ abilities in the job market.
4. MPA Biennial Student Survey (2014): A student survey is administered to MPA Students on a
biennial basis. This survey seeks feedback on our student population and employment, interest
in the MPA program, student support services, core classes, concentrations, value of
coursework, quality of MPA program, course format and scheduling, information and
communication, and overall satisfaction.
5. SPA Triennial Alumni Survey (2013): A survey is administered to all SPA alumni on a triennial
basis. While some questions are presented to all students, the survey is designed so that MPArelated questions are directed to alumni of the MPA program. The survey inquires MPA alumni
about employment status, experience at the SPA, competencies, program quality, student
support services, and demographics. Responses of all MPA graduates as well as recent MPA
graduates are assessed. Recent MPA graduates include those reporting a completion year of
2011, 2012, or 2013. Given the timing of administering the survey, fewer than 5 respondents
completed their degree in 2013.
6. MPA Alumni Employment Survey (2014): Beginning in 2013, a survey of recent MPA
graduates (those graduating in the Summer, Fall, and Spring of the previous academic year) is
administered each January. This survey serves to evaluate whether graduates are employed, and
if so in what sector and type of organization, within six months of graduation.
2
Four Learning Objectives
To assess the four learning objectives, we look to rubric data for each student completing his/her
capstone from the primary reader and the secondary reader. This report looks at rubrics from the
Summer 2013, Fall 2013 and Spring 2014 semesters. The four objectives are:
1) Demonstrate a broad base of knowledge concerning public or nonprofit management, policy,
and research;
2) Use effective critical thinking and analytical skills to solve problems in public and nonprofit
settings in a manner that will improve the public good;
3) Communicate effectively in writing; and
4) Communicate effectively in speaking.
In the Capstone rubrics, the primary instructor (1st reader) and the second reader (SPA faculty) were
asked to assess students on each learning objective.
1=Below proficient
2=Proficient
3=Above proficient
Not every student received a score from each reader in every category, which explains the variance in
the number of students across learning objectives.
Learning Objective 1: Base of Knowledge
“The student’s demonstration of his/her knowledge of public affairs or nonprofit research, theory, and
practice.”
Reader
Term
Avg.
Summer
2013
N=10
2.6
1st Reader
Fall 2013 Spring
2014
N=50
N=48
2.4
2.5
Summer
2013
N=8
2.4
2nd Reader
Fall 2013
N=20
2.5
Spring
2014
N=36
2.4
Learning Objective 2: Critical Thinking
“The student’s demonstration of critical think skills.”
Reader
Term
Avg.
Summer
2013
N=10
2.6
1st Reader
Fall 2013
Spring
2014
N=51
N=48
2.4
2.6
Summer
2013
N=8
2.5
2nd Reader
Fall 2013
Spring 2014
N=19
2.6
N=32
2.5
3
Learning Objective 3: Written Communication
“The student’s demonstration of written communication skills.”
Reader
Term
Summer
2013
N=11
2.5
Avg.
1st Reader
Fall 2013
N=51
2.4
Spring
2014
Summer
2013
N=48
2.6
N=8
2.4
2nd Reader
Fall 2013
N=20
2.6
Spring
2014
N=36
2.4
Learning Objective 4: Oral Communication
“The student’s demonstration of oral communication skills.”
Reader
Term
Avg.
Summer
2013
N=10
2.7
1st Reader
Fall 2013 Spring 2014
N=49
2.5
N=47
2.7
Summer
2013
N=8
2.3
2nd Reader
Fall 2013
Spring 2014
N=19
2.6
N=32
2.5
Learning Objectives Summary
For all four learning objectives, across three semesters, for 1st and 2nd readers, average scores (per
semester by reader) ranged between 2.3 and 2.7. With a score of 2 indicating proficiency, average
scores indicate SPA Capstone students, as a whole, perform above a proficient level for each learning
objective.
Five Competencies
To assess the five competencies, we look to survey data from four sources: The MPA Student Exit
Survey (2013-2014 AY); the MPA Biennial Student Survey (2014); the SPA Triennial Alumni Survey
(2013); and the Capstone Inventory of Competencies. Additionally, we look to questionnaires filled out
during the Capstone rubric process that were distributed with the rubrics for each student completing
his/her Capstone project. Specific questions were analyzed for each competency and are identified
below. The five competencies are:
1)
2)
3)
4)
5)
Lead and manage in public governance;
Participate in the policy process;
Analyze, synthesize, think critically, solve problems, and make decisions;
Articulate and apply a public service perspective; and
Communicate and interact productively with a diverse and changing workforce and citizenry.
4
Competency 1: Lead and Manage
Source
Data
Collection
Sampling
1 = This
category is
both
relevant to
the project
AND was
performed
proficiently.
2 = This
category
was
relevant to
the project,
BUT was
NOT
performed
proficiently.
3 = Not
relevant
to
project.
Capstone
Inventory
“(Student)
understands
and is able to
apply theories
of leadership
to motivate
people, build
teams, and
manage
change.”
Summer
2013
1st Reader
N=10
Fall 2013
1st Reader
N=46
4
1
5
21
2
23
21
1
19
46
4
47
Spring
2014
1st Reader
N=41
Total
N=97
Source
Data
Collection
Sampling
MPA
Student
Survey
“How much
do you
believe your
coursework at
SPA (so far)
has improved
your abilities
and
understanding
in ‘my ability
to be a
leader?’”
Current
SPA
students
N=159
Exiting
SPA
graduates
(Summer
and Fall
2013;
Spring
2014)
N=81
MPA
Exit
Survey
A great
deal
A good
deal
Somewhat
Not
much
32
(20%)
68
(43%)
45
(28%)
Not yet
taken
(student)
/ Not at
all (exit)
10
4
(6%)
(3%)
23
(28%)
37
(46%)
18
(22%)
2
(3%)
1
(1%)
5
Source
Alumni
Survey
Data
Collection
Sampling
“Based on
what you
learned in
the MPA
program,
please rate
your ability
to lead and
manage in
public
governance”
All MPA
Alumni
Respondents
N=146
All MPA
respondents
with 2011,
2012, or
2013
graduation
N=58
No Ability
Basic
Very
Good
Excellent
7
(5%)
46
(32%)
61
(42%)
32
(22%)
2
(3%)
28
(48%)
21
(36%)
7
(12%)
Upon terminating the MPA, 52% of students chose a capstone project that emphasized the lead and
change competency. Of those, 92% successfully demonstrated that competency in the capstone project.
In answer to the question “How much do you believe your coursework at SPA (so far) has improved
your abilities and understanding in ‘my ability to be a leader?’”, 63% of current students and 74% of
graduating students answered either a “great deal” or a “good deal”.
In answer to the question, “Based on what you learned in the MPA program, please rate your ability to
lead and manage in public governance,” 64% of all alumni and 48% of recent alumni answered “very
good” or “excellent.”
6
Competency 2: Participate in the Policy Process
Source
Data
Collection
Sampling
1 = This
category is
both
relevant to
the project
AND was
performed
proficiently.
2 = This
category was
relevant to
the project,
BUT was
NOT
performed
proficiently.
Capstone
Inventory
“(Student)
understands
and applies
tools for
engaging
citizens in the
policy
process.”
Summer
2013
1st Reader
N=11
9
0
2
Fall 2013
1st Reader
N=46
22
2
22
Spring
2014
1st Reader
N=41
23
1
17
Total
N=98
54
3
41
Somewhat
Not
much
Source
Data
Collection
Sampling
MPA
Student
Survey
MPA
Exit
Survey
“How much
do you
believe your
coursework at
SPA (so far)
has improved
your abilities
and
understanding
in ‘my ability
to participate
effectively in
the policy
process?’”
Current SPA
students
N=158
Exiting SPA
graduates
(Summer
and Fall
2013;
Spring
2014)
N=82
A great
deal
A good
deal
3 = Not
relevant
to
project.
33
(21%)
55
(35%)
38
(24%)
14
(9%)
Not yet
taken
(student)
/ Not at
all (exit)
18
(11%)
24
(29%)
33
(40%)
19
(23%)
6
(7%)
0
(0%)
7
Source
Alumni
Survey
Data
Collection
Sampling
“Based on
what you
learned in
the MPA
program,
please rate
your ability
to
participate
in the
policy
process”
All MPA
Alumni
Respondents
N=145
All MPA
respondents
with 2011,
2012, or
2013
graduation
N=58
No
Ability
Basic
Very Good
Excellent
2
(1%)
35
(24%)
71
(49%)
37
(26%)
1
(2%)
17
(29%)
27
(47%)
13
(22%)
Upon terminating the MPA, 58% of students chose a capstone project that emphasized the
participation in the policy process competency. Of those, 95% successfully demonstrated that
competency in the capstone project. In answer to the question “How much do you believe your
coursework at SPA (so far) has improved your abilities and understanding in ‘my ability to participate
effectively in the policy process?’”, 56% of current students and 69% of graduating students answered
either a “great deal” or a “good deal”. In answer to the question, “Based on what you learned in the
MPA program, please rate your ability to participate in the policy process,” 75% of all alumni and 69%
of recent alumni answered “very good” or “excellent.”
8
Competency 3: Analyze, Synthesize, Think Critically, Solve Problems, Make Decisions
Source
Data
Collection
Sampling
Capstone
Inventory
“(Student)
finds and
synthesizes
existing data
to inform
decisions.”
Summer
2013
1st Reader
N=11
10
0
1
Fall 2013
1st Reader
N=46
42
1
3
Spring
2014
1st Reader
N=41
38
3
0
Total
N=98
90
4
4
Source
MPA
Student
Survey
MPA
Exit
Survey
Data
Collection
Sampling
“How much
do you
believe your
coursework at
SPA (so far)
has improved
your abilities
and
understanding
in ‘my ability
to use data to
make
decisions and
solve
problems?’”
Current
SPA
students
N=158
Exiting
SPA
graduates
(Summer
and Fall
2013;
Spring
2014)
N=82
1 = This
category is
both relevant
to the project
AND was
performed
proficiently.
A great
deal
2 = This
3 = Not
category was
relevant to
relevant to the project.
project, BUT
was NOT
performed
proficiently.
A good
deal
Somewhat
Not
much
41
(26%)
69
(44%)
25
(16%)
Not yet
taken
(student)
/ Not at
all (exit)
12
11
(8%)
(7%)
25
(30%)
40
(49%)
13
(16%)
4
(5%)
0
(0%)
9
Source
Alumni
Survey
Data
Collection
Sampling
“Based on
what you
learned in
the MPA
program,
please rate
your ability
to analyze,
synthesize,
think
critically,
solve
problems,
and make
decisions”
All MPA
Alumni
Respondents
N=145
All MPA
respondents
with 2011,
2012, or
2013
graduation
N=58
No Ability
Basic
Very Good
Excellent
1
(1%)
11
(8%)
58
(40%)
75
(52%)
0
(0%)
6
(10%)
22
(38%)
30
(52%)
Upon terminating the MPA, 96% of students chose a capstone project that emphasized the competency
to analyze, synthesize, think critically, solve problems, make decisions. Of those, 96% successfully
demonstrated that competency in the capstone project. In answer to the question “How much do you
believe your coursework at SPA (so far) has improved your abilities and understanding in ‘my ability
to use data to make decisions and solve problems?’”, 70% of current students and 79% of graduating
students answered either a “great deal” or a “good deal”. In answer to the question, “Based on what
you learned in the MPA program, please rate your ability to analyze, synthesize, think critically, solve
problems, and make decisions,” 92% of all alumni and 90% of recent alumni answered “very good” or
“excellent.”
10
Competency 4: Articulate/Apply a Public Service Perspective
Source
Data
Collection
Sampling
1 = This
category is
both
relevant to
the project
AND was
performed
proficiently.
2 = This
category
was
relevant to
the project,
BUT was
NOT
performed
proficiently.
3 = Not
relevant
to
project.
Capstone
Inventory
“(Student)
understands
and adheres to
policies and
practices
advancing
transparency
and
accountability.”
Summer
2013
1st Reader
N=11
5
0
6
Fall 2013
1st Reader
N=46
18
1
27
Spring
2014
1st Reader
N=41
19
3
19
Total
N=98
42
4
52
Source
Data
Collection
Sampling
MPA
Student
Survey
MPA
Exit
Survey
“How much
do you
believe your
coursework at
SPA (so far)
has improved
your abilities
and
understanding
in ‘my ability
to make
ethical
decisions?’”
Current SPA
students
N=157
Exiting SPA
graduates
(Summer
and Fall
2013; Spring
2014)
N=81
A great
deal
A good
deal
Somewhat
Not
much
38
(24%)
58
(37%)
36
(23%)
11
(7%)
Not yet
taken
(student)
/ Not at
all (exit)
14
(9%)
31
(38%)
33
(41%)
12
(15%)
4
(5%)
1
(1%)
11
Source
Alumni
Survey
Data
Collection
Sampling
“Based on
what you
learned in
the MPA
program,
please rate
your ability
to articulate
and apply a
public
service
perspective”
All MPA
Alumni
Respondents
N=145
All MPA
respondents
with 2011,
2012, or
2013
graduation
N=57
No Ability
Basic
Very Good
Excellent
1
(1%)
12
(8%)
67
(46%)
65
(45%)
0
(0%)
6
(11%)
27
(47%)
24
(42%)
Upon terminating the MPA, 47% of students chose a capstone project that emphasized the competency
to articulate and apply a public service. Of those, 91% successfully demonstrated that competency in
the capstone project. In answer to the question “How much do you believe your coursework at SPA (so
far) has improved your abilities and understanding in ‘my ability to make ethical decisions?’”, 61% of
current students and 79% of graduating students answered either a “great deal” or a “good deal”. In
answer to the question, “Based on what you learned in the MPA program, please rate your ability
articulate and apply a public service perspective,” 91% of all alumni and 89% of recent alumni
answered “very good” or “excellent.”
12
Competency 5: Communicate and Interact
Source
Data
Sampling 1 = This
Collection
category is
both
relevant to
the project
AND was
performed
proficiently.
Capstone
Inventory
“(Student)
understands
and
appreciates
the value of
diverse
backgrounds
and
viewpoints in
a democracy.”
2 = This
category
was
relevant to
the project,
BUT was
NOT
performed
proficiently.
3 = Not
relevant
to
project.
Summer
2013
1st Reader
N=11
6
2
3
Fall 2013
1st Reader
N=43
27
1
15
Spring
2014
1st Reader
N=41
30
0
11
Total
N=95
63
3
29
Source
Data
Collection
Sampling
A great
deal
MPA
Student
Survey
“How much do
you believe
your
coursework at
SPA (so far)
has improved
your abilities
and
understanding
in ‘my ability
to understand
and relate to
different
perspectives?’”
Current
SPA
students
N=157
Exiting
SPA
graduates
(Summer
and Fall
2013;
Spring
2014)
N=82
49
(31%)
60
(38%)
31
(20%)
Not yet
taken
(student)
/ Not at
all (exit)
13
4
(8%)
(3%)
27
(33%)
42
(51%)
11
(13%)
1
(1%)
MPA
Exit
Survey
A good
deal
Somewhat
Not
much
1
(1%)
13
Source
Alumni
Survey
Data
Collection
Sampling
“Based on
what you
learned in the
MPA
program,
please rate
your ability
to
communicate
and interact
productively
with a diverse
and changing
workforce
and
citizenry”
All MPA
Alumni
Respondents
N=146
All MPA
respondents
with 2011,
2012, or
2013
graduation
N=58
No Ability
Basic
Very
Good
Excellent
2
(1%)
18
(12%)
61
(42%)
65
(45%)
1
(2%)
7
(12%)
27
(47%)
23
(40%)
Upon terminating the MPA, 69% of students chose a capstone project that emphasized the competency
to communicate and interact productively with a diverse and changing workforce and citizenry. Of
those, 95% successfully demonstrated that competency in the capstone project. In answer to the
question “How much do you believe your coursework at SPA (so far) has improved your abilities and
understanding in ‘my ability to understand and relate to different perspectives?’”, 69% of current
students and 84% of graduating students answered either a “great deal” or a “good deal”. In answer to
the question, “Based on what you learned in the MPA program, please rate your ability to
communicate and interact productively with a diverse and changing workforce and citizenry,” 87% of
all alumni and 87% of recent alumni answered “very good” or “excellent.”
Competency Summary
Based on the capstone inventory data, between 91% and 95% of students are successfully
demonstrating the competency that their capstones emphasize. The strongest performances are to
participate in the policy process; to analyze, synthesize, think critically, solve problems, make
decisions; and to communicate and interact. The weakest performance is to apply a public service
perspective. Current and graduating students demonstrate a marked difference between abilities to
attain competencies, with graduating students consistently demonstrating higher understanding and
ability. This is a logical trend, as students learn in the program, and supports that the program is
influential in teaching the five competencies. Alumni reported being very competent in the areas of
analysis and applying a public service perspective, and less competent at leading and managing as well
as participating in the policy process.
14
Two Outcomes
To assess the two outcomes, we again look to survey data. Specific questions were analyzed for each
outcome and are identified below. Again, the two outcomes are:
1) Graduates will work in professional management or policy positions in public or nonprofit
settings.
2) Graduates will make use of the knowledge and skills they have gained to perform effectively in
their positions.
Outcome 1: Employment
To assess this outcome, we looked at the MPA Alumni Employment Survey. The specific question
asked: “Please choose which description most closely matches your CURRENT employment.”
Employer
Federal government in the U.S.
State, provincial or regional
government in the U.S.
City, county, or other local
government in the U.S.
Foreign government (all levels) or
international quasi-governmental
Nonprofit domestic-oriented
Nonprofit/NGOs internationallyoriented
Private sector – Research/Consulting
Private sector but not
research/consulting
Obtaining further education
Unemployed
Other
N=44
Percent response
9%
14%
Number response
4
6
11%
5
5%
2
16%
5%
7
2
5%
20%
2
9
0%
11%
5%
0
5
2
26 respondents (59%) work in the public or nonprofit setting.
Outcome 2: Making use of Knowledge and Skills
To assess this outcome, we used two survey questions from the MPA Alumni survey.
15
Source
Alumni Survey
Source
Alumni Survey
Data Collection
“Has obtaining
your degree at
SPA helped you
advance your
current position
or secure better
employment?
Scoring
Yes or no.
Data Collection
“In your opinion,
did the MPA
program enhance
your ability to
integrate (high
levels of quality)
into your
professional
work?”
Scoring
No, not at all
Yes, a little
Sample
All MPA alumni
responses.
N=143
All MPA alumni
responses
graduating in
2011, 2012, or
2013
N=55
Results
Yes: 99 (69%)
Sample
All MPA alumni
responses.
N=143
Results
6 (4%)
No: 44 (31%)
Yes: 34 (62%)
No: 21 (38%)
21 (15%)
Yes, somewhat
72 (50%)
Yes, significantly
44 (31%)
3 (5%)
MPA alumni
responses
graduating in
2011, 2012, or
2013
N=55
10 (18%)
28 (51%)
14 (25%)
Generally, results from the alumni survey regarding students’ ability to make use of their knowledge
and skills are consistent whether considering all MPA alumni or only those who have graduated most
recently.
Assessment Report Conclusions
Overall, the assessment of the learning objectives, competencies, and outcomes suggest that our MPA
program is doing well. Some inconsistencies exist between current students and alumni in the strengths
of the curriculum they perceive and exhibit. For instance, students completing their capstones were
weakest in demonstrating applying a public service perspective, but this is one area that alumni report
being the strongest. Generally, our program can do better with the leading and managing and with the
participating in policy process parts of our curriculum. The MPA faculty are reviewing the current
curriculum with an eye for improvement. As part of our stakeholder development, we are also seeking
the input of members of the SPA advisory board and the MPA Student Advisory Committee.
16