Baker College - Master of Business Administration Program Assessment Report 2014-2015
CGS Assessment Report: MBA Program 2014-2015
Table of Contents
Assessment Process Overview ..................................................................................................................... 2
Assessment Process ..................................................................................................................................... 4
Results: Direct Measures of Student Learning ............................................................................................ 5
Results: Indirect Measures of Student Learning ......................................................................................... 7
EBI Alumni Survey ............................................................................................................................. 7
IVY Test.............................................................................................................................................. 9
Results: Key Performance Indicators ........................................................................................................... 9
Progress Report on 2014-2015 Continuous Improvement Action Plan ................................................... 10
Continue to Monitor the Quality of Students in the MBA Program - Completed .......................... 10
Continue to Enhance Presentations Skills of MBA Students - Completed...................................... 11
Continue to Build Teamwork/Collaboration Skills of MBA Students - Completed ......................... 11
Continue to Improve MBA Student Learning Efficiency in Finance - In Progress ........................... 11
Continue to Improve Instructional Quality in the MBA Program - In Progress .............................. 11
To Increase Opportunities for Students to Connect to Practice in the MBA Curriculum - In
Progress................................................................................................................................................... 11
To Ensure Smooth Transition to New Direct Measures Data Collection and Analysis Tool Completed............................................................................................................................................... 11
To Meet New Outcomes Assessment Requirements of IACBE – In Progress ................................. 12
2015-2016 Continuous Improvement Action Plan .................................................................................... 12
Continue to Improve MBA Student Learning Efficiency in Finance ................................................ 12
Continue to Improve Instructional Quality in the MBA Program ................................................... 12
To Increase Opportunities for Students to Connect to Practice in the MBA Curriculum ............... 12
To Prepare for the Quarters to Semesters Conversion .................................................................. 12
To Meet New Outcomes Assessment Requirements of IACBE ....................................................... 12
Expected Results ............................................................................................................................. 13
Appendix A. Center for Graduates Studies Institutional Outcomes......................................................... 14
Appendix B. MBA Program Outcomes....................................................................................................... 15
1
Baker College - Master of Business Administration Program Assessment Report 2014-2015
Assessment Process Overview
The Center for Graduate Studies embraces a philosophy of continuous quality improvement
and requires program administrators to use a variety of robust assessments to ensure that the
stated mission and goals are achieved. Both internal and external assessments are utilized to
monitor and evaluate the graduate program, allocate resources, create professional
development, and update processes as part of the continuous quality improvement cycle.
Specifically, the graduate program assessment process is designed to evaluate data from three
areas: 1) direct measures of student learning outcomes, 2) indirect measures and 3) key
performance indicators.
Faculty members, in collaboration with instructional designers, are responsible for developing
standardized assessment materials to be used within courses. Authentic assessment materials
are designed to evaluate student capabilities as they relate to program and institutional
outcomes. These standardized assessment instruments become a part of the course, and all
faculty members teaching the course are required to administer the instruments. It should be
noted that all standardized assessment instruments are developed with the intent to embed
the assessment process within the course. In this manner, students are not asked to complete
additional assignments or assessments beyond those that are a part of the normal educational
process. This embedding of assessment measures is important to the faculty of the Center for
Graduate Studies, who believe that assessment should be an integral piece of the educational
process, not an addition to it. The assessment materials are designed to support faculty
members in their classroom assessment and evaluation, present students with clear
expectations and performance parameters, and provide students with detailed feedback on
performance as it relates to learning outcomes.
In addition to the direct measures, data are collected through the use of indirect measures,
including surveys of program graduates, employer surveys, and/or accrediting agency reports.
These data are combined with direct measures to complete the assessment data set
Key Performance Indicators (KPI) have been developed to complete the assessment plan. These
KPI are intended to measure programs in relation to priorities that have been set by the
Institution based on our mission and values. The Center has identified the following as KPIs for
evaluating the success of our graduate programs:
•
•
•
•
•
Enrollments
Retention
Graduation rates
Employment rates of graduates
Faculty credentials
These KPIs provide data for analysis and evaluation on metrics beyond teaching and learning.
These metrics provide the primary operational data necessary for evaluating the stability of the
program as well as for planning, budgeting, high level assessment of operations, and how the
2
Baker College - Master of Business Administration Program Assessment Report 2014-2015
program contributes to the mission and guiding principles of the institution. Additionally, these
metrics are compared across graduate programs developing benchmarks, internal targets, and
minimum performance standards.
Annually, the program administrator has the responsibility of compiling the data, discussing and
analyzing the data with the Faculty Council, and collaboratively developing a continuous
improvement plan. The continuous improvement plan is designed to identify the steps
necessary for improving student learning in the designated areas. To address specific findings,
the plan may include identifying actions such as redevelopment of a course, seeking additional
data to clarify student achievement, or requesting alteration of specific assignments or teaching
strategies to improve attainment of learning outcomes. Based on the findings, the plan may
also include operational alterations to such areas as student services or faculty development.
In addition to a review of data collected, the program administrator and the Faculty Council will
undertake an annual review of the program assessment plan to determine the effectiveness of
the plan, and the quality and usefulness of the data collected. As a portion of this annual
review, it is anticipated that the assessment plan for each program will remain a dynamic
document, continuing to evolve as the faculty become more experienced in the process of
program assessment.
3
Baker College - Master of Business Administration Program Assessment Report 2014-2015
Assessment Report
Program: Master of Business Administration (MBA)
Dean: Dr. Na “Lina” Li
Year: 2014-2015
Assessment Process
1. Collect data regarding:
a. Student learning (direct measures/course embedded assessments)
b. Indirect measures (Alumni perceptions regarding learning, faculty, and
curriculum)
c. Key performance indicators
2. Review and analyze data with the following stakeholders:
a. Graduate Faculty Council
b. Advisory Board
c. Center for Graduate Studies Administration
3. Develop a Continuous Improvement Action Plan in collaboration with faculty
4. Submit assessment report:
a. Chief Academic Officer
b. IACBE
c. System Coordinator for Academic Assessment
5. Publish assessment report:
a. Faculty consumption
b. Student consumption
c. Staff and other stakeholders
6. Implement Continuous Improvement Action Plan
7. Review progress on the Continuous Improvement Action Plan of the prior year
assessment report
4
Baker College - Master of Business Administration Program Assessment Report 2014-2015
Results: Direct Measures of Student Learning
The results of the direct measures of student learning, for both Center for Graduate Studies
(CGS) institutional outcomes (Appendix A) and MBA Program outcomes (Appendix B), are
demonstrated in Tables 1-2 and Figures 1-2.
The CGS institutional outcomes assessment results are quite positive. 90% of students or more
performed at a satisfactory or higher level on all outcomes as shown in Table 1 and Figure 1.
This is consistent with the previous years. Student performance on a few outcomes were
slightly higher than last year.
Table 1. Student Performance on CGS Institutional Outcomes 2014-2015
IO1
IO2
IO3
IO4
IO5
IO6
Mean
Exemplary
85.0%
58.0%
69.0%
46.7%
82.7%
59.0%
66.7%
Approaches Below
Satisfactory
Accomplished Satisfactory expectations expectations or above
13.0%
2.0%
0.0%
0.0%
100.0%
25.8%
11.1%
3.4%
2.20%
94.8%
20.7%
8.2%
1.5%
1.0%
98.0%
31.9%
15.6%
3.0%
3.0%
94.1%
16.8%
0.5%
0.0%
0.0%
100.0%
26.2%
10.5%
2.8%
1.4%
95.7%
22.4%
8.0%
1.8%
1.3%
97.1%
Figure 1. Student Performance on CGS Institutional Outcomes 2014-2015
Assessment results on MBA Program outcomes are shown in Table 2 and Figure 2. 90% of
students or more performed at a satisfactory or higher level on all MBA Program outcomes. The
results were similar to the previous years’ regarding PO#1-10, and #12. Indeed, student
performance on a few outcomes were slightly higher than last year. It is worth noting that
student performance on PO#11 (analyze the impact of information systems and technology on
a business and demonstrate the ability to make effective information management decisions)
went above 90% for the first time, which was 92.7%. Further analysis showed that faculty did
5
Baker College - Master of Business Administration Program Assessment Report 2014-2015
use the rubric consistently when evaluating students. Performance regarding this outcome will
continue to be carefully monitored.
Table 2. Student Performance on MBA Program Outcomes 2014-2015
PO1
PO2
PO3
PO4
PO5
PO6
PO7
PO8
PO9
PO10
PO11
PO12
Mean
Exemplary
59.0%
82.7%
66.0%
69.1%
75.2%
69.1%
72.1%
27.4%
46.7%
85.0%
63.2%
33.0%
62.4%
Accomplished
26.2%
16.8%
22.1%
17.5%
16.8%
17.5%
19.3%
41.7%
31.9%
13.0%
20.2%
45.4%
24.0%
Satisfactory
10.5%
0.5%
10.3%
6.5%
5.6%
6.5%
6.2%
25.0%
15.6%
2.0%
9.4%
14.4%
9.4%
Approaches
expectations
2.8%
0.0%
1.5%
4.1%
2.4%
4.1%
1.5%
5.6%
3.0%
0.0%
4.3%
4.3%
2.8%
Below
expectations
1.4%
0.0%
2.8%
0.0%
2.8%
1.0%
0.4%
3.0%
0.0%
3.0%
2.8%
1.6%
Satisfactory
or above
95.7%
100.0%
98.4%
93.1%
97.6%
93.1%
97.5%
94.0%
94.1%
100.0%
92.7%
92.8%
95.8%
Figure 2. Student Performance on MBA Program Outcomes 2014-2015
6
Baker College - Master of Business Administration Program Assessment Report 2014-2015
Results: Indirect Measures of Student Learning
EBI Alumni Survey
The EBI Alumni Survey was again utilized to gain information regarding student perception of
the MBA Program in the 2014-2015 academic year. Sixty five (65) graduates participated in the
survey. Table 3 shows the results of this indirect measurement of the MBA Program outcomes.
EBI provides external comparisons on PO2-8 and 11. Our graduates’ ratings were higher than
the means of national participating institutions on all these outcomes. No comparison was
provided on PO 1, 9 or 10. Our graduates’ ratings met the preset expectations on PO 9 and 10,
but not on PO1 (Communicate using the advanced oral and written communication skills
necessary for success in the business environment).
PO1 includes two components, written and oral communication skills. The average rating of
Baker College MBA graduates on written communication skills was 5.43 on a 7-point Likert
scale, while the oral communication skills rating mean was 4.40. It was the latter that made the
PO1 rating relatively low. As possessing oral communication skills is required by employers, four
oral presentation assignments were designed and implemented in two MBA core courses, MIS
511 (Management Information Systems) and BUS 660 (The Marketing Environment) during
Spring 2013 – Spring 2014. It is believed that these oral presentation opportunities will improve
students’ oral communication skills. However, the effect may not be fully revealed by the EBI
survey in a few years since those who take these classes may not graduate and take the survey
immediately. Measurement result of this PO will continue to be monitored.
PO1
PO2
PO3
PO4
PO5
PO6
Table 3. EBI Survey Results on MBA Program Outcomes (POs)
Measured by
Results
Written and Oral
Average rating of Baker College MBA graduates (4.93) < 5.0 on a
Communication
7-point Likert scale.
Skills questions
Ethics and Legal
Average rating of Baker College MBA graduates (6.15) > national
Issues questions
average of participating institutions (5.65).
Data-Driven
Decision-Making
questions
Domestic and Global
Economies
questions
Critical Thinking and
Problem Solving
questions
Critical Thinking and
Problem Solving
questions
Average rating of Baker College MBA graduates (6.00) > national
average of participating institutions (5.68).
Average rating of Baker College MBA graduates (5.97) > national
average of participating institutions (5.62).
Average rating of Baker College MBA graduates (6.00) > national
average of participating institutions (5.74).
Average rating of Baker College MBA graduates (6.00) > national
average of participating institutions (5.74).
7
Baker College - Master of Business Administration Program Assessment Report 2014-2015
PO7
Financial
Information
questions
Organizational
Behaviors questions
Average rating of Baker College MBA graduates (5.82) > national
average of participating institutions (5.61).
PO9
Multicultural and
Diversity questions
Average rating of Baker College MBA graduates (5.72) > 5.0 on a
7-point Likert scale.
PO10
Professional
Development
questions
Use and Manage
Technology
questions
Average rating of Baker College MBA graduates (1.82) < 2.0 on a
reversed 7-point Likert scale.
PO8
PO11
Average rating of Baker College MBA graduates (5.88) > national
average of participating institutions (5.78).
Average rating of Baker College MBA graduates (5. 90) >
national average of participating institutions (5.22).
In addition, EBI provides external comparisons on various aspects of the program. Figure 3
shows student ratings of the MBA Program in three overarching categories, student overall
satisfaction, overall learning, and overall program effectiveness. All three measures continued
to fall in the more satisfied area, with ratings of 6.25, 6.29, and 6.15 on 1-7 Likert scales. These
ratings were all higher than last year. And they were all significantly higher than the selected six
institutions and all participating institutions.
Figure 3. Baker College MBA Program Overall Ratings
EBI provides comparisons among institutions on 18 factors such as quality of instruction,
breadth of curriculum, aspects of courses, etc. Our MBA program had a statistically higher
mean than the means of the selected six (6) institutions and all participating institutions on
8
Baker College - Master of Business Administration Program Assessment Report 2014-2015
each factor in the 2014-2015 academic year. In particular, student satisfaction on breadth of
curriculum improved from 4.83 in 2013-2014 to 5.33 this year. We will continue to monitor this
factor.
IVY Test
IVY Test was used as an additional indirect measure as previous years. 159 students completed
the assessment this year. The results are shown in Figure 4. Overall, our MBA students
performed lower than the test takers from IACBE institutions and all participating institutions.
In the subject areas, we outperformed the IACBE institutions and all participating institutions in
Strategy. However, our student performance was lower than them in Finance, Financial
Accounting, Managerial Accounting, Management and Marketing.
This was the first time that our students did not outperform or had a tie with the other
institutions since we began to take the test in the 2011-2012 academic year. Two factors might
have contributed to this result. First, when we started to take the IVY Test, we chose 4 hours as
the test time. In contrast, most other schools chose a shorter time. At the beginning of the
2014-2015 academic year, significant changes have been made to the exam. Among those
changes, test time was standardized to 2.5 hours. Many students panicked during the test due
to the short test time. Second, some students experienced technical problems during the test
and had to contact the test hosting company to reset it and retake it. Nevertheless, it is hard to
conclude the lower performance was caused by these two factors or something else, or was
just due to random variation. We will closely monitor this assessment result.
Figure 4. IVY Test Results 2014 - 2015
Results: Key Performance Indicators
The MBA program is a mature program with over 15 years of operational and sustainable
history. Due to the shrinkage of the adult students market and increased competition,
continued growth in size and registrations is unlikely and unexpected. Nevertheless, the
9
Baker College - Master of Business Administration Program Assessment Report 2014-2015
program continues to be the largest graduate program and maintains sufficient registrations to
remain more than viable and productive. Table 4 shows the key performance indicators of the
MBA program for the recent years. The employment and related employment rates of those
who graduated in the 2013 – 2014 academic year were high. Employment data of the 20142015 academic year are not finalized and will be updated as this information becomes
available. The retention and persistence rates in 2014-2015 were lower than previous years.
The graduation rate has been low in two years. These KPIs will be closely monitored.
Table 4. MBA Program Key Performance Indicators
MBA Program
Total New Students
Total Registered Students
Retention Rate
1st Year Persistence Rate
Total Graduates
Graduation Rate
Employment Rate
Related Employment Rate
200809
434
1,346
80.2%
62.2%
377
60.9%
97.5%
92.4%
200910
342
1,156
78.8%
67.6%
307
67.1%
91.0%
95.3%
201011
330
1,085
79.6%
72.0%
284
67.3%
83.9%
83.1%
201112
314
980
79.2%
68.7%
256
67.0%
93.2%
89.7%
201213
263
887
79.8%
63.8%
223
73.4%
93.3%
87.0%
201314
216
744
79.8%
65.7%
225
59.8%
99.5%
93.7%
201415
182
596
55.8%
56.4%
163
59.4%
N/A
N/A
Progress Report on 2014-2015 Continuous Improvement Action Plan
Great progress was made on all the action items in the Continuous Improvement Action Plan
during the 2014-2015 year.
Continue to Monitor the Quality of Students in the MBA Program - Completed
A study was conducted to examine whether the quality of admissions essays is related with
academic performance in the MBA program. All MBA full-time faculty evaluated 50 essays
randomly selected. Dr. Livingston analyzed the data. The study found the following:
1. There is a plausible and modest relationship between applicant scores on the
grammar, form, and style section of the admissions essay and student GPAs.
2. There is a strong relationship between total essay scores and student GPA.
3. There appears to be a plausible but extremely modest relationship between
grammar, form, and style scores on the essay and the likelihood of graduation.
4. There appears to be a plausible but modest relationship between total essay scores
and the likelihood of graduation.
Although many other factors may impact student academic performance, admissions essays
seems to be a factor that is influential and feasible to be assessed prior to admission. Therefore,
it has become a routine practice that admissions essays from all applicants to the MBA program
are evaluated by faculty using the essay rubric. The MBA Dean and the Admission Department
find the evaluations to be very helpful in making admission decisions.
10
Baker College - Master of Business Administration Program Assessment Report 2014-2015
Continue to Enhance Presentations Skills of MBA Students - Completed
The direct measurement instrument for oral communication skills, i.e., the oral presentation
assignment rubric in MIS511, was implemented in Blackboard Clickable Rubric. Corresponding
announcement and training was provided to faculty. Starting in Winter 2015, assessment data
on oral presentation skills have been collected. The assessment result is reported in the Results:
Direct Measures of Student Learning section of this report.
Continue to Build Teamwork/Collaboration Skills of MBA Students - Completed
The direct measurement instrument for teamwork skills, i.e., the team project rubrics in
BUS615, was implemented in Blackboard Clickable Rubric. Corresponding announcement and
training was provided to faculty. Starting in Winter 2015, assessment data on teamwork skills
have been collected. The assessment result is reported in the Results: Direct Measures of
Student Learning section of this report.
Continue to Improve MBA Student Learning Efficiency in Finance - In Progress
The MBA Dean and all faculty who teach BUS 640 (The Financial Environment) met to discuss
student needs and possible strategies to improve learning efficiency in this course. A detailed
course revision plan was developed. The leading faculty of finance revised the BUS 640
curriculum and made it effective in Winter 2015.
Continue to Improve Instructional Quality in the MBA Program - In Progress
Faculty Growth and Evaluation was completed with all MBA full-time and adjunct faculty.
Professional Growth Plans (PGP) have been created with each faculty member. Follow-up of
PGP is ongoing. In addition, quality assurance reports on instructors’ course readiness and
teaching performance from the Instructional Effectiveness Department have been utilized to
identify and address problems on an ongoing base.
To Increase Opportunities for Students to Connect to Practice in the MBA Curriculum - In
Progress
A committee was formed to conduct a survey among current MBA students to understand how
much professional experience they have, whether they prefer the MBA program to expand the
curriculum breadth by providing more opportunities for practical experiences and interaction
with practitioners, and if so, in what formats they would like to welcome these opportunities.
Results were discussed among faculty and advisory board members. Multiple improvement
strategies were proposed. In particular, career service opportunities and advice will be
introduced early in the program. This will be realized in CGS 501 The Graduate Seminar.
To Ensure Smooth Transition to New Direct Measures Data Collection and Analysis Tool Completed
Baker College stopped using Waypoint in the end of Fall 2014. Since Winter 2015, new tools –
Blackboard Clickable Rubric and Blackboard Outcomes, have been employed for collecting
direct measures assessment data. The dean worked closely with the Baker College system
11
Baker College - Master of Business Administration Program Assessment Report 2014-2015
assessment team to transfer all direct measures into the new tools. Training on data collection
was provided to faculty. Direct assessment data from WayPoint and Clickable Rubric in 20142015 were collected, analyzed and reported in the Results: Direct Measures of Student Learning
section of this report.
To Meet New Outcomes Assessment Requirements of IACBE – In Progress
The MBA Program’s accreditation body, IACBE, has new requirements on outcomes
assessment. It requires the development and assessment of major outcomes, broad-based
operational goals, and intended operational outcomes. All these outcomes and goals have been
developed. Measures and criteria of the intended operational outcomes have been identified
and assessment results collected. These achievements have been reported to IACBE.
2015-2016 Continuous Improvement Action Plan
Improvement Strategy:
While the assessment data indicate students are well served in the MBA program, we strive for
continuous improvement. The following strategies will be implemented in the 2015-2016
academic year for this purpose.
Continue to Improve MBA Student Learning Efficiency in Finance
The leading faculty of finance will continue to revise finance curriculum on a rotation base. The
efficacy of the changes will be measured.
Continue to Improve Instructional Quality in the MBA Program
Faculty Growth and Evaluation and follow-up on Professional Growth Plans will be conducted
by the Instructional Effectiveness Department starting in Winter 2016. The MBA Dean will
actively involve in the procedure to help faculty to improve instructional quality on an ongoing
base.
To Increase Opportunities for Students to Connect to Practice in the MBA Curriculum
Career service opportunities and advice will be introduced in CGS 501 The Graduate Seminar.
To Prepare for the Quarters to Semesters Conversion
In order to better serve students, Baker College has decided to convert from quarters to
semesters in Fall 2017. The Dean and faculty will revise step plans, course rotation schedules,
outcome mapping, curricula and any other relevant aspects/documents of the program to
prepare for the conversion.
To Meet New Outcomes Assessment Requirements of IACBE
The outcomes of each MBA major have been developed. Direct and indirect measures of these
outcomes will be identified, developed and implemented in the 2015-2016 academic year.
12
Baker College - Master of Business Administration Program Assessment Report 2014-2015
Expected Results
Results will be shown in direct measures, the EBI survey, the IVY test, and student evaluations
of instructors. It may take several years to see certain results of several projects, as students do
not complete the EBI survey and the IVY test until graduation.
13
Baker College - Master of Business Administration Program Assessment Report 2014-2015
Appendix A. Center for Graduates Studies Institutional Outcomes
Graduates of Baker College’s Center for Graduates Studies will possess:
1. The skills necessary to contribute to their profession through active participation in
scholarly and/or professional activities.
2. The broad-based discipline specific knowledge necessary to manage and advance their
professional career.
3. Data driven decision making skills necessary to produce successful outcomes.
4. An awareness and appreciation for diversity and its impact on their professional
environment.
5. The knowledge and ability to follow and support the ethical standards of their
profession.
6. The communication skills to clearly and proficiently share ideas, knowledge and data
with a wide variety of audiences.
14
Baker College - Master of Business Administration Program Assessment Report 2014-2015
Appendix B. MBA Program Outcomes
Graduates of the MBA program will be able to:
1. Communicate using the advanced oral and written communication skills necessary for success in
the business environment.
2. Apply standards of ethical and legal behaviors in a professional environment.
3. Collect, interpret and analyze existing and/or original research, using quantitative and statistical
tools, and use in the decision making process.
4. Analyze the interrelatedness of market, economic, social and political trends, and their impact
on a global environment.
5. Analyze the strategic planning process, and develop and assess strategic plans.
6. Reason analytically and apply theory across interdisciplinary boundaries to solve problems and
create innovative solutions.
7. Analyze financial reports, risk management strategies and their impact on the decision making
process.
8. Evaluate various leadership strategies and the implications of their use.
9. Discuss problems from diverse perspectives and analyze the impact of individual and cultural
differences on the business environment.
10. Illustrate commitment to personal and professional development, community service and lifelong learning.
11. Analyze the impact of information systems and technology on a business and demonstrate the
ability to make effective information management decisions.
12. Analyze the strategic impact of human resource development and management on a business.
15