Randall Dean Workshops Return On Investment (ROI) Report
Fall 2008 and Spring 2009
Major Findings:
High average overall effectiveness rating: 4.74 – 4.8 on a five point scale
High congruence between planned action steps at end of session and skills that were
actually implemented in the workplace
Strong ROI figure of 302% based on perceptions of time gained from implementation of
new skills
Objectives and Background
In response to stated staff needs (through both formal and informal means), Randall Dean, a
time management consultant, conducted two different workshops for CMU staff in the fall of
2008: one on how to use Outlook to gain productivity, and one on email and office clutter. The
purpose of these workshops was to increase staff efficiency and productivity by reducing the
amount of time used in handling and managing email and other tasks involved with office
organization and scheduling.
Due to a very positive initial response and a substantial waiting list, two additional workshops
were scheduled and conducted in the spring of 2009. Because of the cost and popularity of
these workshops, a deeper level of evaluation was conducted after the first two workshops had
already taken place. In addition to the paper based initial reaction surveys given at the end of
the workshops, an electronic post session survey was designed and distributed to participants
to better measure the impact the training was having in the workplace. One of the specific
purposes of this survey was to isolate how much time the participants gained (that could now
be used for other work) as a direct result of these workshops. This allowed a Return On
Investment (ROI) figure to be calculated, which put into monetary figures what the University
gained in productivity from the money it invested.
Methodology
The five levels of evaluation as described in the table below has become the standard of
reference regarding evaluation strategies for training. Therefore, the terminology of these 5
levels will be used in this report, and used as an outline in its presentation.
1
Characteristics of Evaluation Levels1
Level
Level 1: Reaction and Planned Action
Level 2: Learning
Level 3: Job Applications
Level 4: Business Results
Level 5: Return on Investment
Brief Description
Measures participants’ reactions to the
program and plans for implementation.
Measures skills, knowledge, or attitude
changes.
Measures changes in behavior on the job and
specific applications of the training material.
Measures business impact of the program.
Compares the monetary value of the results
with the costs of the program, usually
expressed as a percentage.
Table 1
Level 1 data was gathered through satisfaction surveys handed out at the end of the sessions
(Appendix 1).
While no explicit attempt was made to obtain Level 2 data (i.e. pre and post knowledge tests),
implicit information concerning what was learned is available. For instance, when a specific skill
is listed as having been implemented, it can be implied that the knowledge and necessary skill
for its implementation has been learned.
Level 3, 4 and 5 data was gathered through an electronic post session survey (Appendix 2). This
survey was sent to the participants of the 2008 workshops in January of 2009, 8 weeks after
their involvement in the last workshop of the fall sessions. This survey was also sent to all of
the staff who participated in the two workshops conducted in 2009 approximately three weeks
after they had attended the workshop.
To calculate the ROI of these workshops, a very conservative approach was used. The only
measure that was chosen to use for this calculation was the time gained at work as a result of
implementing new skills which were learned in the workshop. The following calculations and
assumptions were made:
1. An intentional decision was made to combine the data of the participants from the
two 2008 workshops into one group. The same was done with the participants of
the 2009 workshops. In this way the 2008 participants can be viewed as a group, the
2009 participants can be viewed as a group, or a combination of all of the
participants can be tabulated. This was also done because some of the content from
the two workshops overlapped, and the intent of the evaluation was to view the
impact of the Randall Dean workshops overall, and not to compare the effectiveness
of one to the other. This method created some duplicate people in the data because
some people took both workshops (Optimizing Outlook and Taming the E-Mail
Beast). This amounts to 10 people in 2008 and 6 people in 2009. Therefore, in 2008
1
Phillips, Jack J., 1997. Return On Investment in training and performance improvement programs. Houston, TX:
Gulf Publishing Company.
2
there were 55 unique individuals who took at least one of Randall Dean’s
workshops, while the total attendance for the 2008 workshops was 65. For 2009,
these numbers are 71 unique individuals, and a total attendance of 77. Therefore,
the response rate of the surveys were based on the unique individual count (55 for
2008), while the total cost calculations of the workshops were based on the total
attendance (65 for 2008).
2. The average salary plus benefits was calculated into an hourly rate for all of the 55
participants in 2008 ($33.40/hr) and for the 71 participants in 2009 ( $32.54/hr.).
The benefits factor that was added was 40%.
3. The response rate for the 2008 survey was 27% (15 of 55) and was 37% (26 of 71) for
the 2009 survey. Those who did not answer the follow up survey are assumed to
have no time saving benefits from the training. The ROI figures are therefore based
on 41 of the total 126 who attended the trainings.
4. To isolate the effects of training to a given result, which in this case was time savings
at work, the method of Expert Estimation was employed. The participants of the
workshops were asked in the survey to provide an estimate of how much time they
were saving at work specifically as a result of the skills which they learned in the
Randall Dean workshops. Furthermore, the participants were asked to provide a %
of confidence for the estimate they provided.
5. Participant estimates of time were then reduced by the % of confidence which was
given for his or her estimate (i.e. if a participant estimated 2 hours [120 min.] saved
per week with a 60% confidence rating, 120 min. was multiplied by .60 = 72 min).
6. Monetary productivity gains are calculated for a one year time span, assuming 46
weeks of work time (six weeks were allowed for sick, vacation, and holiday time).
7. Program costs are fully loaded, including the costs of average salaries of all 142
attendees for the 3 hours of the workshop, HR staff time in set up, tear down,
registration, and evaluation, Randall Dean’s costs, and costs of all materials.
8. The formula used for the ROI calculation was:
ROI =
Net Program Benefits (Benefits – Costs)
Program Costs
X 100
This number represents how much productivity has been gained in monetary terms
after the costs have been subtracted out (i.e. an ROI of 200% would mean that for
every $1 invested, $2 was gained in return, after the costs have been removed).
3
Results
Reaction and Planned Action Data – Level 1
Initial responses on the evaluations handed out at the end of the session were very positive.
Scores averaged above a 4.6 on a 5 point Likert Scale in all five areas shown below in Table 2. A
1 indicated Strongly Disagree and a 5 was Strongly Agree.
Initial Participant Response Data
Relevant/Practical
Information
Met
Objectives
Presenter Style
and Delivery
Presenter
Knowledgeable
Overall
Rating
Average 2008 Responses
4.86
4.65
4.87
4.97
4.8
Average 2009 Responses
4.77
4.71
4.78
4.86
4.74
Table 2
On the electronic follow up survey, a more specific question related to relevancy was asked.
How much of what you learned is applicable to your job?
12
10
2008
8
2009
6
Mode = 100
4
Mean = 80.5
2
0
0%
20%
40%
60%
80%
100%
Graph 1
With a mode of 100, and a mean of 80.5, participants strongly indicated that the material was
relevant and applicable to their job.
4
The following narrative question was also asked on the survey:
What specific actions will you do differently on the job as a result of this program?
Please be specific.
82 of the 142 possible people responded (58%) with a planned action. The top five responses
are shown in Table 3.
Total ’08 – ’09
Responses
Clean off
Desk/Get Rid
of Stacks
14
Clean up
Email/Keep
small inbox
11
Organize Work
Area
9
Create More
Folders for
Filing Email
8
3 Minute
Rule/Manage
Email Quickly
8
Table 3
On-the-job Application and Business Impact – Level 3 and 4
To determine if the skills were being applied in the workplace, and to get a view of which ideas
and skills were being used the most, the following narrative question was asked:
What ideas or skills (if any) have you actually implemented back in the workplace?
36 of the 41 participants who completed the survey (88%) specifically listed one or more
items that they had implemented. The five most listed areas are shown in Table 4.
Cleaned up email/
keeping small
inbox
Created
folders for
filing emails
14
12
Total ’08 – ’09
Responses
Using 3
minute rule/
managing
email quickly
11
Using task
features more/
prioritizing work
Got rid of
stacks/
managing
paper files
7
8
Table 4
It is interesting to note the similarity between the intended actions listed on the evaluation
forms handed out at the end of the session, and the actual skills which were implemented.
ROI Calculation based on productivity improvement through time gained – Level 5
Perception of Hours Gained Per Week by Implementation of New Skills
< 30
min
2008 Participant
Responses
2009 Participant
Responses
30 min
1 hour
2 hours
3 hours
4 hours
5 hours
0
0
3
6
3
2
1
2
2
6
7
0
3
1
Table 5
5
Fall 2008 and Spring 2009 Calculations
Monetary figure of time gained in a year
Total perceived hours gained per week by those who completed the survey
Reduced by % of confidence rating given by individuals
Annual amount of hours gained in a year (x 46 weeks)
Times average salary + benefits of attendees (2008 $33.40/hr; 2009 32.54)
Total costs for workshops
Salaries + benefits of participants while in workshop
Randall Dean Costs
Cost of Materials
HR Staff Costs (Registration, Set Up, Evaluation)
2008
2009
37
27
1,242
$41,484
41
30.6
1,407.6
$45,803
$6,513
$3,000
$80
$760
$10,353
$7,517
$3,000
$100
$760
$11,377
2008 Net Program Benefit: $41,484 - $10,353 = $31,131
2009 Net Program Benefit: $45,803 - $11,377 = $34,426
Final ROI Calculations
ROI =
Net Program Benefits
Program Costs
$31,131 + $34,426
$21,730
=
X 100
$65,557 = 3.02 X 100 = 302% ROI
$21,730
For every $1 invested, $3.02 was returned in time productivity gained after costs have
been subtracted. Again, this number is assuming that only the 41 people who returned the
survey are experiencing any gains in time savings, while the remaining 85 who attended the
workshops experienced no gain in time savings. Therefore, the actual number is likely
substantially higher.
6
Intangible Benefits
Though we chose to specifically measure the time gained by participants for the ROI study, this
is certainly not the only benefit that participants and the University have gained through the
training. To discover some of the other intangible benefits, participants were asked in a
narrative question what other impact, besides time savings, the workshop would have on their
work or thinking. 30 of the 41 total who answered the survey responded to this question.
Table 6 shows the top six responses:
Total ’08 – ’09
Responses
Better
organized
(workspace,
projects)
Less
stress/
Greater
peace
Greater
productivity/
Efficiency
Prioritization/
Focusing on
the important
Better
Planning/
Planning
ahead
Improved Mental
Health (more
energy, happier)
15
9
6
5
4
3
Table 6
One person chose not to give an estimate of time saved, but rather shared, “I have no idea how
to estimate, but they [the workshops] have been life changing, invaluable!”
Discussion
Ideas for Additional Support
The final question on the survey asked what kind of support would help the participant to
further use the ideas or tools from the workshop. The following ideas were shared:
Have a book or manual that walks you step by step through some of the techniques
shared for review
Provide a checklist of “must-dos” from the workshop to make sure the participant didn’t
miss anything
10-15 minute refresher of major points covered
The workshop was fast paced – need to have a more thorough workshop with how-to
training materials as a reference
I’d like to take it over again – there was a lot of material crammed into a short time
The ongoing email tips are helpful
Provide a get together of staff on campus to share productivity innovations with one
another. There must be a lot of great ideas out there…
Have Randy come back and keep offering these sessions, and find a way to make them
available for off-campus personnel
Provide a get together of people who have gone through these courses to talk about
how they have implemented what they have learned
7
Lessons Learned
As with projects of any kind, more planning up front will make for a smoother process and
often yield better results. Deciding to do this level of evaluation after two of the workshops
had already taken place made data gathering more difficult. This was a major cause for the
lower than desired return rate, particularly for the 2008 data as the participants were not
aware that a second survey would be coming.
The timing of the electronic surveys sent out varied from 3 weeks after the completion of a
workshop, to approximately 8 weeks. This should be more standardized in future evaluation
efforts. An initial follow up survey could be sent one month after the workshop to measure
which skills had been immediately implemented. Another shorter survey could be sent out
after six months to determine which skills had become a habit and which may have just been a
short term effect.
Furthermore, a carefully laid out communication plan in advance clarifying what kind of results
would be generated and to whom they would be shared would have helped in the report
writing stages.
In future efforts to evaluate to the ROI level, more careful and detailed plans will be made
upfront concerning the data gathering process and the communication plan for how the results
will be reported. Randall Dean will be coming back in the fall of 2009 to conduct these two
workshops again. We will evaluate each of these workshops separately to the ROI level for
comparison purposes.
Conclusion
The Randall Dean workshops were a success from the perspective of all levels of the evaluation.
The participant’s initial reactions were enthusiastic, scoring an average of over 4.6 on a five
point Likert scale for every category. The material was perceived as practical and relevant,
and the presenter as enthusiastic and knowledgeable. 88% of participants shared that they
applied new skills learned in the workshop back in the workplace. As a result, the purpose of
improving the productivity of the staff as measured by time gained was met, with a very strong
ROI figure of 302%.
In addition to these measurable results, the impact of some of the intangible benefits may be
just as significant. The bottom line benefits of a greater sense of control, being able to better
organize and prioritize work, and the reduction of stress may be difficult to objectively
measure, but are too important to be ignored.
In addition, the experience gained in the process of conducting this evaluation will aid in future
evaluation and ROI efforts. These skills will be applied upon Randall Dean’s return in the fall of
2010, when we will make stronger efforts for an increased rate of survey return, and see how
that impacts the ROI results.
8
Appendix 1: End-Of-Session Evaluation
CENTRAL MICHIGAN UNIVERSITY
STRATEGY & ORGANIZATIONAL DEVELOPMENT
118 ROWE HALL
SESSION EVALUATION
Session:_________________________________________
Date:_____________________
Presenter(s):_____________________________________
Please take a minute to provide some feedback on this workshop.
Using the scale below, indicate your level of agreement or disagreement with the following
items:
1
Strongly Disagree
2
Disagree
3
Neutral
4
Agree
1._____
The workshop provided relevant and practical information.
2._____
The workshop met the stated learning objectives.
3._____
The format and variety of activities was appropriate.
4._____
The presenter(s) had an effective style and delivery.
5._____
The presenter(s) was knowledgeable in the subject matter.
6._____
Rate this workshop overall:
Unsatisfactory
1
2
3
4
5
5
Strongly Agree
Excellent
1. What was the most effective aspect of this workshop?
2. What specific actions will you do differently on the job as a result of this program? Please be specific.
3. What aspects of this workshop should be improved (time allocation, facility, type of activities, media,
etc.)?
4. What specific feedback do you have for the presenter(s)?
5. What workshops or topics would you like to see offered?
9
Appendix 2: Post-workshop Online Evaluation
1. Thank you for your time and feedback which will help us to know how effective
these sessions are in helping you in your day to day work. This survey is completely
confidential, and should only take you about 10 minutes to complete. Since Randy
Dean's time management workshops were very similar in purpose, we have decided
to combine them into one evaluation form for your convenience.
Please indicate here which workshop, or both, that you attended:
Optimizing Your Outlook: Time Management Strategies for Busy Outlook Users
Taming Email and Office Clutter Beasts: Key Strategies
1. How much of what you learned in this workshop(s) is applicable to your job?
0%
20%
40%
60%
80%
100%
2. How likely are you to use some of the skills you learned in this workshop(s) on the
job?
0% Won't
use
20% Likely
40% Likely
60% Likely
80% Likely
100%
Already using
new skills from
the workshop
3. What ideas or skills (if any) have you actually implemented back in the
workplace?
4. If you are implementing some of these ideas, please provide an estimate of how
much time you think they will save you, on average, in a given week.
Less than
30 minutes
30
Minutes
One
Hour
Two
Hours
Three
Hours
Four
Hours
Five
Hours
Other (please specify)
3. What confidence do you have in the above estimate, expressed as a percentage?
(0% = no confidence; 100% = certainty)
0%
20%
40%
60%
80%
100%
10
1. What other impact, besides time savings, will the workshop have on your work or
thinking?
2. Are there other ideas that you haven't implemented yet, but plan to? If so, what
are they, and when are you planning to implement them?
3. What kind of support would help you further use the ideas or tools from this
workshop(s)?
Thank you for your time and specific feedback about how Randy Dean's time management
workshops are affecting your work. This will greatly help us in determining the return on
investment for these time management programs. Check out our Open Workshops calendar
for other upcoming workshops you may be interested in: Open Workshops
11