Soderquist College of Business
Measurement and Analysis of Student Learning and Performance - TUG
Performance Indicator
1. Student Learning Results
Definition
A student learning outcome is one that measures a specific competency attainment. Examples of a direct assessment (evidence) of student learning attainment that might be used include: capstone performance, third-party
examination, faculty-designed examination, professional performance, licensure examination). Add these to the description of the measurement instrument in column two:
Direct - Assessing student performance by examining samples of student work
Indirect - Assessing indicators other than student work such as getting feedback from the student or other persons who may provide relevant information.
Formative – An assessment conducted during the student’s education.
Summative – An assessment conducted at the end of the student’s education.
Internal – An assessment instrument that was developed within the business unit.
External – An assessment instrument that was developed outside the business unit.
Comparative – Compare results between classes, between online and on ground classes, Between professors, between programs, between campuses, or compare to external results such as results from the U.S. Department of
Education Research and Statistics, or results from a vendor providing comparable data.
Analysis of Results for Traditional Undergraduate Programs the Soderquist College of Business
Performance Measure
Measurable goal
What is your goal?
New Evaluation Results
Achievement of learning
outcomes as measured by
evaluations conducted by two
separate groups - a business
advisory board and internship
supervisors - for two separate,
but related, types of student
work. Desired goal: comparable
results between two evaluation
groups
What is your measurement instrument
or process?
Do not use grades.
Current Results
What are your current results?
Analysis of Results
Action Taken or Improvement
made
What did you learn from the What did you improve or what
results?
is your next step?
Insert Graphs or Tables of Resulting Trends
(3-5 data points preferred)
(Indicate type of instrument) direct,
formative, internal, comparative
Evaluation via survey of each group.
External advisory board evaluates
outcomes as demonstrated in the
Strategic Management Capstone project
whereas internship supervisors evaluate
outcomes based on how a student does as
an intern on the job. Capstone evaluation
is based on agreement of criteria related
to each outcome; internship evaluation is
based on a performance scale. NOTE:
students may not be the same between
the two groups but are representative of
students in the groups. These surveys are
external summative assessments for
which the results are presented in a
comparative manner.
Except in the area of quantitative skills,
evaluations of student outcomes are very
comparable (within 3-5% of each other)
between the two groups of evaluators.
Advisory board evaluations indicate only
75% of those who evaluated students
agreed that those students possess
quantitative skills whereas Intern
supervisors rated quantitative skills at a
90%. Again, Advisory Board evaluations
are based on agreement with statements
whereas internship evaluations are based
on performance.
With the difference in
evaluation results of
quantitative scores, it appears
that either the capstone
project doesn't provide
opportunities for using
quantitative skills OR
evaluators don't have enough
information to assess
quantitative skills OR interns
are given more opportunity to
build on those skills while on
the job.
We will look at the Strategic
Management capstone project
and determine if there are ways to
improve the quantitative
component of that course.
2015-16 Comparison of Capstone Project
vs Internship Evaluations
120%
100%
80%
60%
40%
20%
0%
Comm
Skills
Crit Thkg
Bus Skills
Skills
Formative evaluation by faculty of artifacts
submitted by students via Taskstream.
Rubrics assessing criteria related to each
outcome are used in the evaluation and a
report showing overall results for the past
three years was generated by Taskstream.
This is a direct, formative, internal
assessment that provides comparative
data.
Although the goal of 80% was not
reached in every area in 2014-15, student
evaluation scores increased in every area
from 2014-15 to 2015-16. This
information was not available in
Taskstream prior to Fall 2014.
Although there was an
improvement in quantitative
skills from 2014-15 to 201516, that particular outcome
received the lowest scores.
These results are consistent
with the results shown above.
In addition to looking at ways to
improve the quantitative
component of the capstone
project, we will look into how we
can address this particular
learning outcome in other courses
as well.
Quant
Skills
Advisory Board
(Agreement)
92%
89%
94%
89%
94%
75%
Intern Supervisors
(Performance)
89%
89%
87%
89%
97%
90%
Advisory Board (Agreement)
Achievement of learning
outcomes as measured by faculty
assessment in Taskstream.
Desired rubric average for each
of these six student learning
outcomes: 80%
Ldrshp & Ethics
Teams Underst
Skills
anding
Intern Supervisors (Performance)
TUG Student Learning Outcomes Assessment
90%
85%
80%
75%
70%
65%
Business
Skills and
Knowledge
Comm.
Skills
Problem
Solving
Skills
2014-15
78%
79%
83%
87%
74%
2015-16
80%
81%
85%
88%
81%
2014-15
2015-16
Leadership
Quant. Skills
Skills
Evalution Results Continued from 2015 QA Report
Demonstrated ability to think
analytically and logically
Desired agreement: 80%
Evaluation by external advisory board
(summative assessment)--percent
agreement that students have
demonstrated this ability throughout
course capstone project (Strategic
Management course).
Prior to 2012 results indicate an
acceptable performance. 2012 seems to
be an anomaly as the percentage
increased well above 80% in Fall 2013.
There was a slight dip in Fall 2014 though still above 80% - but the average
score went up again in Fall 2015.
These results show continued
improvement on this specific
outcome. They also reveal
that 2012 was an anomaly in
the results.
We believe this is a critical skill
we need to continue to emphasize
within our courses and monitor the
results.
Ability to Think Analytically & Logically
100%
95%
90%
85%
80%
75%
70%
Fall 2009 Fall 2010 Fall 2011 Fall 2012 Fall 2013 Fall 2014 Fall 2015
Demonstrated ability to apply
Evaluation by external advisory board
accounting and financial concepts (summative assessment)--percent
Desired agreement: 80%
agreement that students have
demonstrated this ability throughout
course capstone project (Strategic
Management course)
Demonstrated ability to show
ethical behavior
Desired agreement: 80%
Evaluation by external advisory board
(summative assessment)--percent
agreement that students have
demonstrated this ability throughout
course capstone project (Strategic
Management course)
The results since 2009 indicate that we
are consistently below our desired
results. However, results do indicate
positive movement in the direction of the
desired 80%.
With several years of results indicated in
this table, it is clear to see that the results
are far above the level required to meet
acceptable performance. It also indicates
that the drop in the Fall of 2012 was an
anomaly and that the last two years'
results have risen and continue to rise.
Overall these results show
continued improvement, with
the trend being upward. This
is still not where we want to
be in this area. With that said,
it should be noted that in Fall
2015 17% of the respondents
indicated neither agree or
disagree and only 8%
disagreed that students were
able to apply accounting and
financial concepts.
One of two professors dedicated
to Accounting classes resigned at
the end of the 2015-16 academic
year, but the other will take on
more responsibility teaching lower
level classes, which will hopefully
build a better foundation for these
skills.
The results validate that our
students are very successful
in demonstrating ethical
behavior.
What we will be working on is
further integrating ethics across
our curriculum as well as
stabilizing the instruction of this
course. Until the past two years,
the instructor had changed each
year which may account for the
varied results however positive
they are.
Knowledge of Accounting and Finance
100%
80%
60%
40%
20%
0%
Fall 2009 Fall 2010 Fall 2011 Fall 2012 Fall 2013 Fall 2014 Fall 2015
Ethical Behavior
100%
95%
90%
85%
80%
75%
70%
Fall 2009 Fall 2010 Fall 2011 Fall 2012 Fall 2013 Fall 2014 Fall 2015
Soderquist College of Business
Measurement and Analysis of Student Learning and Performance - Degree Completion
Performance Indicator
Definition
1. Student Learning Results
A student learning outcome is one that measures a specific competency attainment. Examples of a direct assessment (evidence) of student learning attainment that might be used include: capstone performance, third-party
examination, faculty-designed examination, professional performance, licensure examination). Add these to the description of the measurement instrument in column two:
Direct - Assessing student performance by examining samples of student work
Indirect - Assessing indicators other than student work such as getting feedback from the student or other persons who may provide relevant information.
Formative – An assessment conducted during the student’s education.
Summative – An assessment conducted at the end of the student’s education.
Internal – An assessment instrument that was developed within the business unit.
External – An assessment instrument that was developed outside the business unit.
Comparative – Compare results between classes, between online and on ground classes, Between professors, between programs, between campuses, or compare to external results such as results from the U.S. Department of
Education Research and Statistics, or results from a vendor providing comparable data.
Performance Measure
What is your measurement instrument
or process?
Do not use grades.
Analysis of Results for Degree Completion Organizational Management Program
Measurable goal
Current Results
What are your current results?
What is your goal?
(Indicate type of instrument) direct,
formative, internal, comparative
Demonstrated ability to think
creatively, critically and
integratively, to effectively make
decisions, to communicate
effectively, to use empowered
leadership, to use business skills
effectively and to understand
Christian worldview. Desired
rubric average for each of these
six student learning outcomes:
80%
Formative evaluation by faculty of artifacts Students' average scores for all learning
submitted by students via Taskstream.
outcomes exceeded our goal of 80%
Rubrics assessing criteria related to each each of the three years.
outcome are used in the evaluation and a
report showing overall results for the past
three years was generated by Taskstream.
This is a direct, formative, internal
assessment that provides comparative
data.
Analysis of Results
Action Taken or Improvement
made
What did you learn from the What did you improve or what
results?
is your next step?
Although the scores indicate a
high level of achievement,
concerns exist over the
validity of the scores as a
reflection of actual student
performance.
Concerns with a possible lack of
rigor as well as other concerns
with the OM program led to the
decision to replace the OM
degree with a new degree in
Organizational Leadership
effective Fall 2016. Despite OM
entering a teach-out phase,
however, lessons can be learned
from the assessment process in
OM and applied to other DC
programs in the future.
Insert Graphs or Tables of Resulting Trends
preferred)
(3-5 data points
Organizational Management
Student Learning Outcome Assessment
Results
1
0.95
0.9
0.85
0.8
0.75
2013-14
2014-15
2015-16
Soderquist College of Business
Measurement and Analysis of Student Learning and Performance - Graduate
Performance Indicator
Definition
1. Student Learning Results
A student learning outcome is one that measures a specific competency attainment. Examples of a direct assessment (evidence) of student learning attainment that might be used include: capstone performance, third-party
examination, faculty-designed examination, professional performance, licensure examination). Add these to the description of the measurement instrument in column two:
Direct - Assessing student performance by examining samples of student work
Indirect - Assessing indicators other than student work such as getting feedback from the student or other persons who may provide relevant information.
Formative – An assessment conducted during the student’s education.
Summative – An assessment conducted at the end of the student’s education.
Internal – An assessment instrument that was developed within the business unit.
External – An assessment instrument that was developed outside the business unit.
Comparative – Compare results between classes, between online and on ground classes, Between professors, between programs, between campuses, or compare to external results such as results from the U.S. Department of
Education Research and Statistics, or results from a vendor providing comparable data.
Performance Measure
What is your measurement instrument
or process?
Do not use grades.
Analysis of Results for Graduate Business Programs
Measurable goal
(Indicate type of instrument) direct,
formative, internal, comparative
Perceived ability to effectively
This evaluation was created to allow
communicate upon completion of students to assess their own abilities
a JBU graduate business
within the graduate business student
program. Goal is that 75% of
outcomes. It is an indirect summative
students will assess their skills as assessment.
average, above average or
expert ability.
Current Results
What are your current results?
Analysis of Results
Action Taken or Improvement
made
What did you learn from the What did you improve or what
results?
is your next step?
Insert Graphs or Tables of Resulting Trends
(3-5 data points preferred)
What is your goal?
Perceived ability to analyze
complex situations, identify
relevant issues, apply
appropriate concepts and skills
and develop sound solutions
upon completion of a JBU
graduate business program.
Goal is that 75% of students will
assess their skills as average,
above average or expert ability.
This evaluation was created to allow
students to assess their own abilities
within the graduate business student
outcomes. It is an indirect summative
assessment.
With the exception of academic year
2015-16, results continue to trend in a
positive direction with all graduate
students claiming at least an average
ability to effectively communicate though
some clearly have more confidence in
their abilities than others. The number of
respondents in the 2015-16 year was
very low, perhaps skewing the data for
this year.
Results continue to trend in a positive
direction with all graduate students
claiming at least an average ability to
think critically and strategically though
some clearly have more confidence in
their abilities than others.
As stated in the 2015 QA
report, there was some
concern about the way
questions were framed when
gathering student learning and
performance results using this
assessment tool. There is
also concern that selfassessment may not be the
best measure of overall
assessment.
See above
In an effort to improve overall
student learning and assessment,
the graduate programs underwent
a major review process during the
2015-16 academic year. The
review resulted in a proposal
recommending a revision of the
current curriculum as well as the
addition of two concentrations to
the MBA. Learning outcomes
were also streamlined and plans
were made to evaluate the
achievement of those outcomes
by incorporating artifacts for
assessment in Taskstream for a
few select courses.
SCOB Graduate Business Students
Self Assessment of Effective Communication
80%
60%
40%
20%
0%
2011-12
2012-13
2013-14
2014-15
0%
3%
0%
0%
50%
Average Ability
19.00%
11.67%
7.70%
16.65%
0.0%
Above Average Ability
54.70%
56.67%
71.80%
33.35%
25.0%
Expert Ability
26.33%
28%
20.50%
50%
25.0%
Some Ability
2015-16
See above.
SCOB Graduate Students
Self Assessment of Critical/Strategic Thinking
80%
60%
40%
20%
0%
2011-12
2012-13
2013-14
2014-15
0%
0%
0%
0%
0%
18.33%
15.00%
19.90%
33.35%
25.0%
Above Average Ability 62.46%
58.87%
68.55%
16.65%
50.0%
26%
11.55%
50%
25.0%
Some Ability
Average Ability
Expert Ability
19.12%
2015-16
Perceived ability to understand
the implications associated with
ethical leadership and decision
making upon completion of a JBU
graduate business program.
Goal is that 75% of students will
assess their skills as average,
above average or expert ability.
This evaluation was created to allow
students to assess their own abilities
within the graduate business student
outcomes. It is an indirect summative
assessment.
Results continue to trend in a positive
direction with all graduate students
claiming at least an average ability to
understand the importance of ethical
decision making though some clearly
have more confidence in their abilities
than others.
See above
See above.
SCOB Graduate Business Students
Self Assessment of Ethical Decision Making
80%
60%
40%
20%
0%
2011-12
2012-13
2013-14
2014-15
0%
0%
8%
0%
0.0%
Average Ability
2.57%
2.77%
3.85%
0.00%
25.0%
Above Average Ability
67.90%
62.23%
75.65%
33.35%
50.0%
Expert Ability
29.53%
35%
12.20%
67%
25.0%
Some Ability
Perceived ability to perform
advanced research, analyze
information resources and apply
appropriate research
methodology upon completion of
a JBU graduate business
program. Goal is that 75% of
students will assess their skills as
average, above average or
expert ability.
Perceived ability to understand
the historical evolution of
leadership theory upon
completion of a JBU graduate
business program. Goal is that
75% of students will assess their
skills as average, above average
or expert ability.
This evaluation was created to allow
students to assess their own abilities
within the graduate business student
outcomes. It is an indirect summative
assessment.
This evaluation was created to allow
students to assess their own abilities
within the graduate business student
outcomes. It is an indirect summative
assessment.
With the exception of academic year
2015-16, results continue to trend in a
positive direction with all graduate
students claiming at least an average
ability to perform research and analyze
information though some clearly have
more confidence in their abilities than
others. With that said, this is the only
outcome for which a handful of students
claimed less than average ability every
year except for one. The number of
respondents in the 2015-16 year was
very low, perhaps skewing the data for
this year.
See above
See above.
SCOB Graduate Business Students
Self Assessment of Info Lit/Research Skills
80%
60%
40%
20%
0%
2011-12
2012-13
2013-14
2014-15
2015-16
8%
6%
8%
0%
50.0%
Average Ability
38.47%
13.90%
31.45%
16.65%
0.0%
Above Average Ability
36.30%
71.13%
52.60%
33.35%
25.0%
Expert Ability
17.27%
9%
7.70%
50%
25.0%
Some Ability
Results continue to trend in a positive
direction with all graduate students
claiming at least an average ability to
understand leadership theory and
practice though some clearly have less
confidence in their abilities than others
with the number of those claiming
average ability exceeding those claiming
expert ability over the last two years.
SCOB Graduate Business Student
Self Assessment of Leadership Theory/Practice
80%
60%
40%
20%
0%
2011-12
2012-13
2013-14
2014-15
3%
7%
8%
0%
0.0%
Average Ability
18.17%
8.33%
11.55%
16.65%
50.0%
Above Average Ability
60.83%
67.80%
67.95%
16.65%
25.0%
Expert Ability
18.30%
17%
12.20%
67%
25.0%
Some Ability
Perceived ability to apply
leadership theory to teamwork
upon completion of a JBU
graduate business program.
Goal is that 75% of students will
assess their skills as average,
above average or expert ability.
This evaluation was created to allow
students to assess their own abilities
within the graduate business student
outcomes. It is an indirect summative
assessment.
Results continue to trend in a positive
direction with all graduate students
claiming at least an average ability to
apply leadership theory to team work
though some clearly have more
confidence in their abilities than others.
2015-16
See above
2015-16
See above.
SCOB Graduate Business Students
Self Assessment of Collaboration/Teamwork
100%
80%
60%
40%
20%
0%
2011-12
2012-13
2013-14
2014-15
2015-16
0%
0%
4%
0%
0%
Average Ability
20.10%
9.40%
7.70%
33.50%
0%
Above Average Ability
64.00%
71.10%
76.25%
16.00%
50%
Expert Ability
16.00%
19%
12.20%
50%
50%
Some Ability
Perceived ability to understand
diverse and complex ideas from
the perspective of a Christian
worldview upon completion of a
JBU graduate business program.
Goal is that 75% of students will
assess their skills as average,
above average or expert ability.
This evaluation was created to allow
students to assess their own abilities
within the graduate business student
outcomes. It is an indirect summative
assessment.
Results continue to trend in a positive
direction with all graduate students
claiming at least an average ability to
understand Christian worldview though
some clearly have more confidence in
their abilities than others.
See above
See above.
SCOB Graduate Business Student
Self Assessment of Christian Worldview
100%
80%
60%
40%
20%
0%
2011-12
2012-13
2013-14
2014-15
2015-16
0%
0%
0%
0%
0%
Average Ability
12.27%
11.00%
3.85%
33.35%
0%
Above Average Ability
71.90%
73.86%
88.45%
16.50%
50%
Expert Ability
14.23%
15%
7.70%
50%
50%
Some Ability