How to Address the
Challenging ROI Issues
April 14, 2014 – 10:00 AM – 11:15 AM
Jack J. Phillips, Ph.D.
Chairman, ROI Institute
April 13-16, 2014
Indianapolis, Indiana
www.ispi.org/ac2014
HOW TO ADDRESS THE
THREE MOST CHALLENGING
ROI ISSUES
Jack J. Phillips, Ph.D.
Chairman, ROI Institute, Inc.
Dr. Jack J. Phillips is a world-renowned expert on accountability, measurement, and evaluation. Phillips provides consulting
services for Fortune 500 companies and major global organizations. The author or editor of more than fifty books, he
conducts workshops and presents at conferences throughout the world.
Phillips has received several awards for his books and work. On three occasions, Meeting News named him one of the 25
Most Powerful People in the Meetings and Events Industry, based on his work on ROI. The Society for Human Resource
Management presented him an award for one of his books and honored a Phillips ROI study with its highest award for
creativity. The American Society for Training and Development gave him its highest award, Distinguished Contribution to
Workplace Learning and Development for his work on ROI. His work has been featured in the Wall Street Journal,
BusinessWeek, and Fortune magazine. He has been interviewed by several television programs, including CNN.
His expertise in measurement and evaluation is based on more than 27 years of corporate experience in the aerospace,
textile, metals, construction materials, and banking industries. Dr. Phillips has served as training and development manager at
two Fortune 500 firms, as senior human resource officer at two firms, as president of a regional bank, and as management
professor at a major state university.
Objectives
After completing this session, participants should perceive the content to be:
 Relevant to their work
 Immediately applicable
Participants will be able to:
 Explain the benefits of using return on investment (ROI).
 Describe the steps in the ROI Methodology.
 Explain the three most challenging areas in an ROI study.
 Identify the principal uses of ROI data.
Copyright © 2014 ROI Institute, Inc.
No part of this may be reproduced, stored in a retrieval system, or transmitted in any form or by a means without written permission
Phone: 205-678-8101
Email: [email protected]
Web site: www.roiinstitute.net
Follow, Like or Visit us:
The “Show Me” Evolution From Executives
Term
Issue
Show Me!
Collect Impact Data
Show Me the Money!
And Convert Data to
Money
Show Me the Real
Money!
And Isolate the Effects of the
Project
Show Me the Real
Money, and Make me
Believe it!!
And Compare the Money
to the Cost of the Project
Why ROI Now?






ROI is the ultimate measure
ROI has been the elusive measure
ROI has a rich history of application
Operating managers understand and relate to ROI
ROI builds excitement among stakeholders
ROI is a top executive requirement
The ROI Methodology Captures Six Types of Measures






Reaction and Planned Action
Learning
Application
Business Impact
Return on Investment
Intangible Measures
……..and includes a technique to isolate the effects of the program or solution
1
Status of Measurement
Level
Current
Status*
Measurement Category
Goal
Inputs/Indicators
0
Measures the number of programs,
participants, audience, costs, and efficiencies
100%
100%
Measures reaction to, and satisfaction with,
the experience, contents, and value of
program
100%
100%
Must use simple
learning measures
Measures what participants learned in the
30 – 40% 80 – 90%
program – information, knowledge, skills, and
contacts (takes-away from the program)
54% †
Application
3
Need more follow-up
Measures progress after the program – the
use of information, knowledge, skills, and
contacts
10%
30%
31% †
10%
This is the
connection to
business impact
Business Impact
4
Measures changes in business impact
variables such as output, quality, time, and
cost-linked to the program
5%
14.4% †
ROI
5
Need more focus on
content and
perceived value
79% †
Learning
2
This is being
accomplished now
100% †
Reaction and Planned Action
1
Comments About
Status
Compares the monetary benefits of the
business impact measures to the costs of the
program.
* Percent of Programs Evaluated At This Level
† Best Practice Benchmarking (user for 5 plus years)
» Add your numbers in each box
Exercise:
What concerns do you have about these percentages?
2
1%
5%
The ultimate level of
evaluation
4.3% †
Five Levels of Measurement - Examples
Level 0 Input and Indicators
 Number of programs
 Participants
 Hours
 Requests



Efficiencies
Costs
Time to Deliver
Level 1 Reaction and Planned Action
 Relevance*
 Importance*
 Usefulness
 Appeal
 Emotion
 Brevity






Uniqueness
Concreteness
New Information*
Motivation
Appropriateness
Intent to Use*
Level 2 Learning
 Information
 Knowledge
 Understanding
 Capability




Contacts
Confidence
Perceptions
Skills
Level 3 Application
 Use of Information
 Use of Knowledge
 Use of Skill
 Completion of Actions
 Completion of Tasks
 Implementation of Ideas






Following the Policy
Use of Procedure
Use of Regulation
Success with Application
Barriers
Enablers
Level 4 Business Impact
 Productivity
 Quality
 Sales
 Errors
 Incidents
 Re-Work
 Efficiency







Compliance Discrepancies
Citizen Complaints
Costs Employee Engagement
Employee Retention
Service Delivery
Cycle Time
Customer Satisfaction
Level 5 Return on Investment
 ROI (%)
 Benefit/Cost Ratio
 Payback Period
Intangibles
…. includes a technique to isolate the effects of
the project.
* Correlates with Application
Exercise:
Any surprises here?
___________________________________________________________________________________________
___________________________________________________________________________________________
___________________________________________________________________________________________
3
The ROI Methodology ™
Capture Costs
Of Program
Data Collection
Evaluation Planning
LEVEL 1:
Reaction and
Planned Action
Develop/
Review
Objectives of
Program
Develop
Evaluation
Plans and
Baseline Data
Collect Data
During
Program
Implementation
LEVEL 2:
Learning
Data Analysis
Reporting
LEVEL 3:
Application
Collect Data
After
Program
Implementation
Isolate the
Effects of
Program
Convert Data
to Monetary
Value
Calculate
the Return On
Investment
Generate
Impact
Study
LEVEL 5: ROI
LEVEL 4:
Business Impact
Identify
Intangibles
Intangible Benefits
Evaluation Framework
ROI Calculation
ROI =
Net Project Benefits
Project Costs
Cost of project $230,000
Benefits of project (1st year) $430,000
ROI =
$430,000-$230,000
$230,000
Level
Measurement Focus
1. Reaction & Planned
Action
Measures participant reaction to the program and
captures planned action.
2. Learning
Measures changes in knowledge and skills.
3. Application
Measures implementation, actions, and changes in
behavior on the job.
4. Business Impact
Measures changes in business impact variables.
5. Return on
Investment
4
Compares monetary benefits of the impact of the
program.
= 0.87 x 100 = 87%
Guiding Principles
The ROI Process
An evaluation system must have five elements.
1. When conducting a higher-level evaluation, collect data at
lower levels.
2. When planning a higher level evaluation, the previous
level of evaluation is not required to be comprehensive.
3. When collecting and analyzing data, use only the most
credible sources.
4. When analyzing data, select the most conservative
alternatives for calculations.
5. Use at least one method to isolate the effects of the
program or project.
6. If no improvement data are available for a population or
from a specific source, assume that no improvement has
occurred.
7. Adjust estimates of improvements for the potential error of
the estimates.
8. Avoid use of extreme data items and unsupported claims
when calculating ROI calculations.
9. Use only the first year of annual benefits in the ROI analysis
of short-term solutions.
10. Fully load all costs of the solution, project, or program when
analyzing ROI.
11. Intangible measures are defined as measures that are
purposely not converted to monetary values.
12. Communicate the results of the ROI Methodology to all key
stakeholders.
An Evaluation
Framework
Applications
and Practice
Implementation
A Process
Model
Operating Standards
and Philosophy
Criteria for Selecting Programs for Level 4 and Level 5
Evaluation
Benchmarking*
•
•
•
•
•
•
•
•
Life cycle of the program
Linkage of program to operational goals and issues
Importance of program to strategic objectives
Executive interest in the evaluation
Cost of the program
Visibility of the program
Size of target audience
Investment of time required
*Survey of Users, N = 235
5
14%
29%
50%
48%
52%
45%
6%
7%
Program Alignment
V Model
Start Here
Payoff Needs
End Here
5
Business Needs
ROI Objectives
4
Performance Needs
5
Impact Objectives
3
4
Application Objectives
3
ROI
Impact
Application
Initial
Measurement and
Evaluation
Analysis
Learning Needs
Preference Needs
Business Alignment and Forecasting
2
Learning Objectives
1
Reaction Objectives
2
1
Learning
Reaction
Project
The ROI Process Model
6
Challenge #1:
Matching Exercise
Level
A. Survey
E. Focus Groups
B. Test
F. Observation
C. Questionnaire
G. Performance Records
D. Interview
Write the instrument’s letter in the box to the right of each question. Also, indicate
the level of evaluation pursued (1, 2, 3, or 4).
1. Customer service representatives have learned to resolve customer complaints in the
most effective manner. An integral part of the program required customer service
representatives to follow a series of planned steps to resolve the complaint, using
empathy and listening skills. As part of the evaluation, the HR staff must determine
the extent to which participants are actually utilizing the newly acquired skills.
Instrument
For each of the following situations, please indicate the most appropriate type of instrument being used to collect
data needed in the program’s evaluation. Select from these types:

2. Intact team members are involved a conflict resolution program where they acquired
skills to resolve conflicts and disputes among themselves. Team members in this
work group have a high degree of interaction and some responsibilities include
checking the work of others. There had been an unusually high level of friction with
displays of open conflicts in the group. In the program, participants learned how to
deal with these issues and work together as a smooth operating team. The HR staff
needs to collect information about the group’s progress with skills, ideally in an
environment where there is an opportunity for group members to listen to comments
from others.

3. Technicians participate in an e-learning program on basic mathematics and are
required to achieve a pre-determined level of competency in mathematics after
completing the program. The HR staff measures the level of mathematical ability
before and after the program.

4. The front desk staff at a major hotel has participated in a program to teach them how
to use a new reservation system that is being installed. As part of the evaluation, it is
important to obtain reactions to the program and capture planned actions and a
forecast of success..

5. A company has implemented a new compensation plan in which the employees share
in the overall profits of the company. Employees have attended a roll-out meeting
where they have the opportunity to learn how the program works and what is required
of them to make it successful. As part of the evaluation, management is interested in
finding out what the employees think about the new plan after attending the briefing.

6. Sales representatives have a new commission program designed to improve sales.
One objective of the program is to improve sales volume and the HR staff must
determine exactly what increase was achieved by each individual since the program
was conducted.

7. Supervisors attended a problem-solving program, where they learned a logical
approach to solving significant problems facing their work units. As a part of the
program’s evaluation, the HR staff needs feedback from participants concerning their
use of the acquired skills. The staff thinks there is a possibility of a success story
here and will need to probe for details.

7
Action Plan
Name:
Objective:
John Mathews
Instructor Signature:
Follow-Up Date
Reduce Weekly Absenteeism Rate for Team
Improvement Measure:
Absenteeism Rate
Evaluation Period:
Current Performance
Action Steps
1. Meet with team to discuss reasons for absenteeism –
using problem solving skills..
2. Review absenteeism records for each employee –
look for trends and patterns..
3. Counsel with “problem employees” to correct habits
and explore opportunities for improvement..
4. Conduct a brief “performance discussion” with an
employee returning to work after an unplanned
absence.
5. Provide recognition to employees who have perfect
attendance.
6. Follow-up with each discussion and discuss
improvement or lack of improvement and plan other
action.
7. Monitor improvement and provide recognition when
appropriate.
1 September
March to September
8%
Target Performance
5%
Analysis
A. What is the unit of measure? One Absence
10 March
B. What is the value (cost) of one unit? $ 41.00
20 March
C. How did you arrive at this value?
Standard Value
D. How much did the measure change during the evaluation period?
(monthly value) 2.5%
E. What percent of this change was actually caused by this program?
31 March
65%
F. What level of confidence do you place on the above information?
(100% = Certainty and 0% - No Confidence)
80%
Intangible Benefits: Less Stress, Greater Job Satisfaction
Comments: Great Program – it kept me on track with this problem
8
Increasing Questionnaire Response Rates




























Provide advance communication about the questionnaire.
Clearly communicate the reason for the questionnaire.
Indicate who will see the results of the questionnaire.
Show how the data will be integrated with other data.
Keep the questionnaire simple and as brief as possible.
Keep questionnaire responses anonymous – or at least confidential.
Make it easy to respond; include a self-addressed, stamped envelope/e-mail.
Use the local manager to distribute the questionnaires, show support, and encourage response.
If appropriate, let the target audience know that they are part of a carefully selected sample.
Use one or two follow-up reminders.
Have the introduction letter signed by a top executive.
Enclose a giveaway item with the questionnaire (pen, money, etc.).
Provide an incentive (or chance of incentive) for quick response.
Send a summary of results to target audience.
Distribute questionnaire to a captive audience.
Consider an alternative distribution channel, such as e-mail.
Have a third party gather and analyze data.
Communicate the time limit for submitting responses.
Consider paying for the time it takes to complete the questionnaire.
Review the questionnaire at the end of the formal session.
Carefully select the survey sample.
Allow completion of the survey during work hours.
Add emotional appeal.
Design questionnaire to attract attention, with a professional format.
Let participants know what actions will be taken with the data.
Provide options to respond (paper, email, web-site).
Use a local coordinator to help distribute and collect questionnaires.
Frame questions so participants can respond appropriately and make the questions relevant.
Improving Post Program Data Collection







Plan early: before program is conducted
Communicate expectations
Secure commitment to provide
Build in the data collection tools
Use at least 12 techniques in
Share results
Use the data
9
Challenge No. 2:
Isolating the Effects of the Program
Benchmarking Data
Method
Control Groups
Trend Line Analysis
Forecasting Methods
Participant Estimates
Manager Estimates
Sr. Management Estimates
Expert Input
Customer Input
1.
2.
3.
4.
5.
6.
7.
8.
Frequency
Survey of Users, N = 235
Control Group Design
Control Group
M1
Experimental
Group
M1
M2
Program
M2
Use of Control Groups Example Retention Solution
at the Federal Information Agency
An opportunity to participate in a master’s degree program at agency expense and agency time
•
•
•
•
One hundred high-potential employees chosen for program for a three-year master’s program in
information science
Experimental group of one hundred were involved in the program; another one hundred in control
group were not involved
Observed employee turnover performance of both groups during the same time
Neither group is aware of the control group arrangement
How would you select the control group?
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
10
Use of Trend Line Analysis
1.85% Pre Program Six-Month Average
Fraud Program Conducted
Fraud
Incident
Rates
Projected Average
— Using Pre Data
as a Base 14.5%
2%
7% Post
Program SixMonth Average
J
F
M
A
M
J
J
A
S
O
N
D
J
MONTHS
Example of Estimation
1%
Performance Improvement: Time to Process Visas =15 minutes
Factor that Influenced
Improvement
Confidence
Expressed as a
Percent
Percent of Improvement
Caused By
HR Project
Technology Change
Procedure Change
Other
Total
60%
15%
25%
___%
100%
80%
70%
60%
___%
Adjusted
Percent of
Improvement
Caused By
48%
10.5%
15%
___%
Using Estimates to Isolate the Effects
•
•
•
•
•
•
•
Describe the task and the process.
Explain why the information was needed and how it will be used.
Ask participants to identify any other factors that may have contributed to the increase.
Have participants discuss the linkage between each factor and the specific output measure.
Provide participants with any additional information needed to estimate the contribution of each
factor.
Obtain the actual estimate of the contribution of each factor. The total must be 100%.
Obtain the confidence level from each employee for the estimate for each factor (100%=certainty;
0%=no confidence).
The Power of Estimates
•
•
•
•
•
•
Research
Comparison with other methods
Handling objections
11
Management reactions
Participant reaction
Use as a last resort
Matching Exercise
Instructions: For each of the following situations, please indicate the method used to isolate the effects of the
program. Select from these methods:
A. Control group
B. Trend line analysis
C. Forecasting
D. Participant’s estimate
E. Use of customer input
F. Expert’s estimate
Write the letter in the box to reflect the method used.
Situation
1. A manufacturing company has recently implemented a new incentive plan to boost the
sales for client partners. Just as the plan was implemented, the company increased its
promotional budget for each product line. Both the sales incentive plan and the sales
promotion have driven an increase in sales. It appears that no other factors have
contributed to this increase. There is a mathematical relationship between the promotional
budget and the sales increase based on historical data. This equation is used to predict the
sales increase based on the promotional budget increase. This forecast is compared with
actual figures to isolate the impact of the sales incentive program.
2. Absenteeism for bus drivers in a large metropolitan area has been deteriorating for some
time. An HR program was implemented to include a no-fault absenteeism policy and a
change in the selection process. After the program was conducted, the absenteeism rate
decreased. It appears that no other influences have contributed to this decrease. The preprogram absenteeism data are very stable and a trend is projected in the post period to
compare with the actual figures. The difference in the two showed the contribution of the
project.
Method


3. An agent-training program in a real estate firm was designed to increase listings.
Customers are providing their rationale for deciding to list a home with a particular agent.
Listings increased three months after the program was completed. While there are many
factors that caused the increase, one factor was the quality of the presentation made by the
agent, which was the basis of the program. This information was used to understand the
impact of the agent-training program on the actual number of houses listed.

4. A new wellness and fitness center has caused this energy company’s healthcare
expenditures to decrease one year after the center was opened. Given this amount of
decrease, several experts were assembled who understand why healthcare costs have
changed. These individuals are asked to explain all the contributing factors and isolate the
effects of the wellness and fitness center on that measure.



5. A large automobile company implemented a sales consulting process on a pilot basis.
Twelve dealerships were used in the initial pilot program. A comparison group was
selected to judge performance along several measures: sales volume, economy in the
market, the sales vs. service mix, incentives provided to sales staff, and the quality rating of
the dealership. The difference in the two groups showed the impact of the sales consulting.
6. In a leadership development program for a biotech company, the participating managers
were asked to provide details on the impact of the use of the leadership skills, using actual
data in the work unit. As part of the exercise, these participants estimated the percent of
improvement directly related to the leadership development program.
12
Challenge No. 3:
Matching Exercise
Instructions: For each of the following situations, please indicate the method used to convert data to
money. Select from these methods:
A.
B.
C.
D.
E.
F.
G.
H.
I.
J.
Profit/savings from output (standard value)
Cost of quality (standard value)
Employee time as compensation (standard value)
Historical costs/savings from records
Expert input
External database
Linking with other measures
Participant estimation
Management estimation
Estimation from HR staff
Write the letter in the box to reflect the method used.
Situation
1. The Veteran’s Administration was experiencing a high turnover rate of nurses. A new HR
program was designed to reduce this turnover. To obtain a value of one voluntary
turnover, the internet was used to find a study in healthcare that showed the average cost
(fully loaded) of replacing a nurse. This number, expressed as a percent of salary, was
used in the calculation.
2. A new program for couriers at DHL was designed to reduce the number of re-packaging
errors. This error occurs when a package is damaged by the couriers and has to be
repackaged before delivery. The quality office in Brussels had previously determined the
standard cost for a repackaging error. This amount was used to develop the total
monetary value for error reduction.
3. Mid-level managers at an electric utility were involved in a time management program
designed to help managers save time. Each manager estimated the number of hours
saved each week directly attributable to this program. The value used for each hour
saved was the total annual salary for those managers adjusted for employee benefits and
calculated on an hourly basis.
4. An Australian government agency was implementing a new HR program designed to
reduce the number of stress claims for employees. The employees who worked with
angry and upset people were suffering from extreme stress on the job and were filing
claims. To obtain the monetary value for a stress claim, the medical and health staff
provided an average value for one claim based on their expertise of managing stress
claims for several years.
5. A new customer call center program at a home appliance company was designed to
reduce the number of calls that were escalated to the next level of management. The
individuals involved in the program were the immediate supervisors of the employees
taking the calls. To determine the cost of a call escalation, the participants (supervisors)
estimated the cost attached to each of these calls.
6. A pharmaceutical company was implementing a new ethics program for all employees.
While there were several outcomes from this program, one in particular measure was
expense account violations. To obtain the average cost of an expense account violation,
all of the violations for a two-month period were taken directly from the records and
divided by the total violations.
13
Method






7. Employee engagement data are collected for a global computer company. A new HR
program was designed to improve engagement scores. To place a value on a change in
engagement score, the staff examined the correlations between engagement scores and
employee turnover in different job groups. This correlation analysis is part of the human
capital management system. As engagement scores improved, voluntary turnover
decreased. The corresponding cost savings of the turnover reduction was used as the
value for changes in the engagement score.

8. A small equipment manufacturing company was interested in reducing absenteeism. The
program was implemented and the cost of one absence was needed for the monetary
impact. The HR staff member who conducted the study estimated the cost to be $300 per
day.

9. A major retail store chain was anxious to reduce the number of customer complaints. A
new program was implemented and the cost of a complaint was needed. To obtain the
cost of one complaint, the management of the customer service area and the Customer
Care Vice President estimated the average cost of one complaint.

Typical Intangible Benefits
10. Wachovia Bank implemented an advanced negotiation program where commercial
bankers increased revenue from new and existing clients. The outcome of the program
was increased revenue in specific product lines. To calculate the monetary value of a
sale, the revenue amount was multiplied by the profit margin for the product line, which
was considered to be a standard value for the organization.
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Adaptability
Awards
Brand awareness
Career minded
Caring
Collaboration
Communication
Commitment
Conflicts
Cooperation
Corporate social responsibility
Creativity
Culture
Customer complaints
Customer response time
Customer satisfaction
14
Decisiveness
Employee complaints
Engagement
Execution
Image
Innovation
Job satisfaction
Leadership
Networking
Organizational climate
Organizational commitment
Partnering
Reputation
Resilience
Stress
Talent
Teamwork

ROI Is Reported In Two Ways
Program Benefits
Program Costs
Benefit Cost Ratio (BCR) =
ROI (%) =
Net Program Benefits
Program Costs
X 100
Exercise:
Benefits = $500,000
Cost = $400,000
What is the ROI?
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Setting the ROI Objective
What’s a great ROI? It depends on the objectives
 Set the value as with other investments – e.g.15%
 Set the value slightly above other investments – e.g. 25%
 Set at breakeven – e.g.0%
 Set at client expectations –e.g. 30%
ROI Red Flags






Not using a balanced profile of data
Failure to isolate the effects of the program
Overstating the monetary benefits stream
Leaving out some costs
Not being conservative
Not being consistent
ROI Methodology : The Payoff










Align projects to business needs
Show contributions of selected projects
Earn respect of senior management/administrators
Build staff morale
Justify/defend budgets
Improve support for projects
Enhance design and implementation processes
Identify inefficient projects that need to be redesigned or eliminated
Identify successful projects that can be implemented in other areas
Earn a “seat at the table”
Exercise:
What are your payoff opportunities?
________________________________________________________________________________
________________________________________________________________________________
15
Additional Resources
To continue to build your expertise in ROI, additional resources may be desired. Here is a list of most of
the recent books on the ROI Methodology.
Victor V. Buzachero, Jack Phillips, Patricia Pulliam Phillips, Zack L. Phillips: Measuring ROI in
Healthcare, New York: McGraw-Hill. 2013
Jack Phillips, Patricia Pulliam Phillips, Lizette Zuniga. Measuring the Success of Organizational
Development, Alexandria: ASTD 2013
Jack J. Phillips, Patricia Pulliam Phillips, Rachel Robinson: Measuring the Success of Sales Training: A
step-by-step Guide for Measuring Impact and Calculating ROI. Alexandria, 2013
Bruce Aaron, Jack J. Phillips, Patricia Pulliam Phillips. Survey Basics: Alexandria: ASTD 2013
Jack Phillips and Patricia Pulliam Phillips. The Consultant’s Scorecard: Tracking ROI and Bottom-Line
Impact of Consulting Projects, 2nd Edition. New York: McGraw-Hill. 2011.
Patricia Pulliam Phillips. The Bottomline on ROI. 2nd Edition, King of Prussia: HRDQ Press, 2012.
Jack J. Phillips and Patricia Pulliam Phillips. Proving The Value of HR: How and Why to Measure ROI,
2nd edition, Alexandria: SHRM, 2012.
Patricia Pulliam Phillips, Jack J. Phillips, Lisa Ann Edwards. Measuring the Success of Coaching: A
Step-by-Step Guide for Measuring Impact and Calculating ROI. Alexandria: ASTD and ROI Institute, Inc.
2012.
Jack J. Phillips, Patricia Pulliam Phillips, Rebecca L. Ray. Measuring Leadership Development: Quantify
Your Program’s Impact and ROI on Organizational Performance. New York: McGraw Hill. 2012.
Patricia Pulliam Phillips and Jack J. Phillips. 10 Steps to Successful Business Alignment. Alexandria:
ASTD and ROI Institute, Inc. 2012.
Patricia Pulliam Phillips and Jack J. Phillips. The Green Scorecard: Measuring the ROI in Sustainability
Projects. Boston: Nicholas Brealey, 2011.
Patricia Pulliam Phillips and Jack J. Phillips. Proving the Value of HR: ROI Case Studies, 2nd ED.
Birmingham, AL: ROI Institute, 2010.
Patricia Pulliam Phillips (ed). Measuring and Evaluating Training. Alexandria: American Society for
Training and Development, 2010.
Patricia Pulliam Phillips. ASTD Handbook of Measuring and Evaluating Training, ASTD Press, 2010.
Jack J. Phillips and Patricia Pulliam Phillips Proving the Value of HR: ROI Case Studies, Second Edition,
Birmingham: ROI Institute, Inc. 2010.
Jack J. Phillips and Patricia Pulliam Phillips. Measuring for Success: What CEOs Really Think About
Learning Investments. Alexandria: ASTD, 2010.
Jack J. Phillips. The Consultant’s Guide to Results-Driven Business Proposals: How to Write Proposals
that Forecast Impact and ROI. New York: McGraw Hill, 2010.
16
Jack J. Phillips. Measuring for Success: What CEOs Really Think About Learning Investments.
Nashville: ASTD Press, 2010.
Jack J. Phillips and Patricia Pulliam Phillips. The Consultant’s Guide to Results-Driven Business
Proposals, New York: McGraw-Hill, 2010.
Jack J. Phillips and Lisa Edwards. Managing Talent Retention: An ROI Approach, San Francisco:
Pfeiffer, 2009.
Jack J. Phillips and Patricia Pulliam Phillips. Beyond Learning Objectives: Develop Measurable
Objectives That Link To The Bottom Line, Alexandria: ASTD, 2008.
Patricia Pulliam Phillips and Jack J. Phillips. The Measurement and Evaluation Series. ROI
Fundamentals: Why and When to Measure Return on Investment; Data Collection: Planning For and
Collecting All Types of Data; Isolation of Results: Defining the Impact of the Program; Data Conversion:
Calculating the Monetary Benefits, Costs and ROI: Evaluating at the Ultimate Level; Communication and
Implementation: Sustaining the Practice, San Francisco: Pfeiffer, 2008.
Patricia Pulliam Phillips and Jack J. Phillips. ROI In Action Casebook, San Francisco: Pfeiffer, 2008.
Jack J. Phillips and Patricia Pulliam Phillips. The Value of Learning: How Organizations Capture Value
and ROI and Translate It into Support, Improvement, and Funds, San Francisco: Pfeiffer, 2007.
Jack J. Phillips. Show Me the Money: How to Determine ROI in People, Projects and Programs, BerrettKoehler, San Francisco, CA, 2007.
Patricia Pulliam Phillips, Jack J. Phillips, Ron Drew Stone, and Holly Burkett. The ROI Field Book:
Strategies for Implementing ROI in HR and Training, Burlington: Elsevier, 2007.
Jack J. Phillips. Investing in Your Company’s Human Capital: Strategies to Avoid Spending Too Little or
Too Much, New York: AMACOM, 2005.
Patricia Pulliam Phillips and Jack J. Phillips. Return on Investment (ROI) Basics: A Complete, How-to
Guide to Help You: Understand and Apply Basic Principles and Practices; Select Appropriate Programs
To Measure; Communicate Results and Sustain Momentum, Alexandria: ASTD, 2005.
Jack J. Phillips and Patricia Pulliam Phillips. ROI at Work: Best-Practice Case Studies from the Real
World, Alexandria: ASTD, 2005.
Jack J. Phillips, Patricia Pulliam Phillips, and Toni Krucky Hodges. Make Training Evaluation Work: Show
Value and Communicate Results; Select the Right Model and Find Resources; Get Management Buy-In
and Overcome Resistance, Alexandria: ASTD, 2004.
Jack J. Phillips. Return on Investment in Training and Performance Improvement Programs, 2nd Edition,
Woburn, MA: Butterworth-Heinemann, 2003.
Patricia Pulliam Phillips, Editor; Jack J. Phillips, Series Editor. In Action: Measuring Intellectual Capital,
Alexandria: ASTD, 2002.
17
The ROI Fact Sheet
Origin/Development/History
 The ROI Methodology™ was developed by Dr. Jack J. Phillips in the 1970s, refined through application and use in the 1980s, and implemented
globally during the 1990s.
 First impact study – 1973, Measuring the ROI in a Cooperative Education Program, for Lockheed-Martin
 First public presentation on the methodology – 1978, ASTD Annual Conference
 First book published to include methodology – 1983, Handbook of Training Evaluation and Measurement Methods, Gulf Publishing (this was the
first USA book on training evaluation)
 First one-day public workshop –1991, Birmingham, Alabama
 First two-day public workshop –1992, Johannesburg, South Africa
 First case study book published – 1994, Measuring Return on Investment, ASTD
 First international partnership established – 1994, Indonesia
 First public certification workshop – 1995, Nashville, Tennessee
 ROI Network organized - 1996
 First ROI Network Conference –1997, New Orleans, Louisiana
 First international ROI Network Conference – 2002, Toronto, Canada
 First ROI in Government Conference – 2003, Gulfport, Mississippi, Co-sponsored by the University of Southern Mississippi
 First ROI software release – 2003, Knowledge Advisors
 Distinguished contribution to workplace learning and performance awarded by ASTD to Jack Phillips for the work on ROI - 2005
 ROI Certification offered as part of Master’s and Ph.D. degree – Capella University, 2006
 ROI Methodology adopted by the United Nations for system implementation- 2008
 One hundred books published with the ROI Institute founders as authors or editors- 2010
 Five thousand participants in ROI Certification- 2010
 A record of 35 public, live certification workshops conducted in a year- 2010
th
 ASTD celebrates 40 book written or edited by Jack and Patti Phillips- 2012
th
th
 Handbook of Training Evaluation and Measurement Methods celebrates 30 anniversary with a 4 edition- 2013
th
 ROI Institute celebrates 20 anniversary- 2013
Use
 More than 5,000 organizations are using the ROI Methodology, through planned implementation
 4,000 organizations have formally implemented the methodology through ROI Certification™ conducted by the ROI Institute
 Approximately 5,000 impact studies are conducted annually in learning and development and human resources
 At least 300 public sector governmental units are using the methodology
 ROI implementation was first pursued in manufacturing, then moved to service, healthcare, non-profits, governments, and is now in educational
systems
Applications
Typical applications include:
Human Resources/Human Capital
Training/Learning/Development
Leadership/Coaching/Mentoring
Knowledge Management Organization
Consulting/Development
Change Management
Policies/Procedures/Processes
Technology/Systems/IT
Green Projects/Sustainability Projects
Talent Retention Solutions Project
Management Solutions Quality/Six
Sigma/Lean Engineering
Meetings/Events/Conferences
Marketing/Advertising
Communication/Public Relations
Public Policy/Social Programs
Risk Management/Ethics/Compliance
Healthcare Initiatives
Wellness and Fitness Programs
Recognition/Incentives/Engagement
Safety and Health Programs
Phone: 205-678-8101
18
Copyright©2014, ROI Institute, Inc.
Email: [email protected]
The ROI Fact Sheet
Articles and Publicity
 More than 100 articles have been published on the ROI Methodology in major publications in 30 countries
 The ROI Methodology has been a cover story on at least 15 publications, magazines, and journals
 At least 100 interviews in major global business and professional publications
 More than 25 radio and TV interviews in different countries
Fact Sheet
Books
 50 books have been published on the ROI Methodology and its application
(www.roiinstitute.net)
nd
 Primary reference – Return on Investment in Training and Performance Improvement Projects, 2 Edition, Jack J. Phillips,
Butterworth-Heinemann, Woburn, MA, 2003 (originally published in 1997)
 Award winning book – Bottomline on ROI, Patricia Pulliam Phillips, CEP Press, 2002 (received
ISPI award) General application – Show Me the Money, Jack J. Phillips and Patricia Pulliam
Phillips, Berrett-Koehler, 2007
 Most comprehensive work – Measurement and Evaluation Series, Jack J. Phillips and Patricia Pulliam Phillips,
Pfieffer, 2008
Case Studies
 More than 200 case studies published in books, journals, and industry publications
 Four-volume set published by ASTD in 1994, 1997, 2001, and 2005
 First public sector case book – 2002, published jointly by the International Personnel and Management Association and the
American Society for Training and Development
 First International case book – 2005, Ireland published by Skillnets
 International case studies under development in 12 countries
Workshops (One-Day, Two-Day, and Three-Day)
 Approximately 200 one-day workshops conducted with more than
8,000 participants
 Approximately 500 two-day workshops conducted with more than 25,000 specialists and managers attending (offered in almost every major
international city)
 Routine schedules of one-day, two-day, and three-day workshops offered in the USA by ASTD and through partners around the world
ROI Certification™
 Five-day workshop plus two work products lead to certification for ROI implementation
 More than 5,000 professionals have attended certification, representing more than 3,000 organizations in at least 50 countries
Certifications offered routinely about 25 times per year both internally and publicly by the ROI Institute (www.roiinstitute.net)
 On-line and self-study options for certification are available (www.roiinstitute.net)
Global Implementation
 First implementation of the ROI Methodology outside the USA – 1992, South Africa
 First certification in non-English language – 1995, Italy
 Implementation is accomplished through partners in various countries
 Implementation is currently occurring in 51 countries, with additional implementations planned in other countries
 Books published in more than 30 languages
 12 international case study books in development or in the planning stages
Phone: 205-678-8101
19
Email: [email protected]
Copyright©2014, ROI Institute, Inc.
Jack J. Phillips, Ph.D.
Dr. Jack J. Phillips is a world-renowned expert on accountability, measurement, and evaluation. Phillips
provides consulting services for Fortune 500 companies and major global organizations. The author or editor
of more than fifty books, he conducts workshops and presents at conferences throughout the world.
Phillips has received several awards for his books and work. On three occasions, Meeting News named him
one of the 25 Most Powerful People in the Meetings and Events Industry, based on his work on ROI. The
Society for Human Resource Management presented him an award for one of his books and honored a
Phillips ROI study with its highest award for creativity. The American Society for Training and Development
gave him its highest award, Distinguished Contribution to Workplace Learning and Development for his work
on ROI. His work has been featured in the Wall Street Journal, BusinessWeek, and Fortune magazine. He
has been interviewed by several television programs, including CNN. Phillips served as President of the
International Society for Performance Improvement, 2012-2013.
His expertise in measurement and evaluation is based on more than 27 years of corporate experience in the
aerospace, textile, metals, construction materials, and banking industries. Dr. Phillips has served as training
and development manager at two Fortune 500 firms, as senior human resource officer at two firms, as
president of a regional bank, and as management professor at a major state university.
This background led Dr. Phillips to develop the ROI Methodologya revolutionary process that provides
bottom-line figures and accountability for all types of learning, performance improvement, human resource,
technology, and public policy programs.
Dr. Phillips regularly consults with clients in manufacturing, service, and government organizations in 44
countries in North and South America, Europe, Africa, Australia, and Asia.
Phillips and his wife, Dr. Patti P. Phillips, have recently published books such as 10 Steps to Successful
Business Alignment (ASTD Press, 2011); Measuring the Success of Coaching (ASTD Press, 2012);
Measuring Leadership Development: Quantify your Program’s Impact and ROI on Organizational
Performance (McGraw-Hill, 2012); The Green Scorecard: Measuring the Return on Investment in
Sustainability Initiatives (Nicholas Brealey, 2011); and Project Management ROI (John Wiley, 2011). They
also recently served as authors and series editors for the Measurement and Evaluation Series published by
Pfeiffer (2008), which includes a six-book series on the ROI Methodology and a companion book of 14 bestpractice case studies. Other books recently authored by Phillips include ROI for Technology Projects:
Measuring and Delivering Value (Butterworth-Heinemann, 2008); Return on Investment in Meetings and
Events: Tools and Techniques to Measure the Success of all Types of Meetings and Events (ButterworthHeinemann, 2008); Show Me the Money: How to Determine ROI in People, Projects, and Programs (BerrettKoehler, 2007); The Value of Learning (Pfeiffer, 2007); How to Build a Successful Consulting Practice
(McGraw-Hill, 2006); Investing in Your Company’s Human Capital: Strategies to Avoid Spending Too Much or
Too Little (Amacom, 2005); Proving the Value of HR: How and Why to Measure ROI (SHRM, 2005); The
Leadership Scorecard (Elsevier Butterworth-Heinemann, 2004); Managing Employee Retention (Elsevier
Butterworth-Heinemann, 2003); Return on Investment in Training and Performance Improvement Programs,
2nd ed. (Elsevier Butterworth-Heinemann, 2003); The Project Management Scorecard, (Elsevier ButterworthHeinemann, 2002); How to Measure Training Results (McGraw-Hill, 2002); The Human Resources Scorecard:
Measuring the Return on Investment (Elsevier Butterworth-Heinemann, 2001); The Consultant’s Scorecard
(McGraw-Hill, 2000); and Performance Analysis and Consulting (ASTD, 2000). Phillips served as series
editor for ASTD’s In Action casebook series, an ambitious publishing project featuring 30 titles. He currently
serves as series editor for Elsevier Butterworth-Heinemann’s Improving Human Performance series.
Dr. Phillips has undergraduate degrees in electrical engineering, physics, and mathematics; a master’s degree
in Decision Sciences from Georgia State University; and a Ph.D. in Human Resource Management from the
University of Alabama. He has served on the boards of several private businesses—including two NASDAQ
companies—and several nonprofits and associations, including the American Society for Training and
Development and the National Management Association. He is chairman of the ROI Institute, Inc., and can be
reached at (205) 678-8101, or by e-mail at [email protected].
20