Return on Investment
In Information Technology:
A Guide for Managers
Anthony M. Cresswell
Center for Technology in Government
University at Albany, SUNY
187 Wolf Road, Suite 301
Albany, NY 12205
Phone: (518) 442-3892
Fax: (518) 442-3886
E-mail: [email protected]
www.ctg.albany.edu
August 2004
©2004 Center for Technology in Government
The Center grants permission to reprint this document provided this cover page is included.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Acknowledgments
T
his report was developed on a foundation of previous work of the Center for Technology
in Government and valuable assistance from several staff members. Sharon Dawes,
Theresa Pardo, Donna Canestraro, and Meghan Cook made important contributions to early
versions. Natalie Helbig, Dubravka Juraga provided excellent suggestions and edits on the
final version. Mark LaVigne helped with editing as well, and also worked with Paula Hauser
on the design of the printed document. The research on costs and returns of investment
in Web site conversion, reported here as a case study, is based on the technical work of
Derek Werthmuller and James Costello, and the assistance of Carrie Schneider and
Yi-Jung Wu.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Table of Contents
Executive Summary
1
Chapter One
5 ROI and the Need for Smart IT Investment Decisions
6 An approach to understanding and using ROI analysis
7 Understanding strategic objectives in ROI design
8 Defining and measuring the costs and returns from IT investments
9 Understanding the enterprise: technology in the business context
12 Understanding ROI decisions in their political and policy context
Chapter Two
12
Political risk factors
13
Organizational risk factors
13
Business process risks
13
Technology risk factors
14 Methods of ROI Analysis for IT
14 Issues of time and scale
14 Importance of business process analysis
15
Benefits of process modeling
16 Methods for understanding the business process
16
Descriptive models
18
Use of analytical or formal models
18
Types of formal modeling approaches
22 Measuring costs and returns
22
Financial metrics and government accounting
23
Measuring costs and cost-effectiveness
24
Efficiency measures
24
Impact measures
25
How time perspectives change the measurements
25
The problem of choosing the project or investment life cycle
25 Risk analysis in the public sector
Appendix A: Case 1 28 Reducing the Cost of Web Site Development and Maintenance
29
Advantages of a dynamic Web site
29
Data sources
30
Cost estimation
31
Benefits
33
Summary of costs and returns
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Table of Contents
Appendix B: Case 2 35 ROI for Data Integration in Health and Human Services
35 Project goals and the context of State-Level welfare reform
35
Project rationale
35
Investment in improved welfare administration
36
Project methods
36
ROI framework
37
ROI results for the Welfare Reform Related Technology Fund
38
Potential risks in benefit estimates
Appendix C: Case 3 40 Social ROI
40 The investment philanthropy approach
Appendix D
40
The social return on investment method
43
Measuring value and returns
43
Limitations in the SROI approach
45 Additional Resources
45 Enterprise architecture
45 Stakeholder analysis
45 Business process analysis and modeling
45 Formal modeling
45
Agent based modeling
46
UML modeling
46
Workflow modeling
46 Operations research
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Executive Summary
T
oday more than ever, government
decision makers must make the most
of scarce resources and at the same time
respond to ever-increasing demands for
improved performance and new technology.
These competing demands generate close
scrutiny of proposals for new information
technology (IT) investments. What’s more,
high profile IT system failures have raised
concerns about why these investments
so often fail to live up to expectations. As
a result, many IT investment planning
processes now require some analysis of
the costs and returns expected from that
proposed investment. Unfortunately, public
sector managers often lack models that can
guide them through such analyses. This
Guide is offered to help fill that gap.
The Guide provides that help by presenting
a practical approach to understanding
what ROI analysis can and cannot do.
A meaningful return on investment (ROI)
analysis in information technology is a little
like saying you want to live a healthier
lifestyle. Like lifestyle changes, ROI
analysis is not just a single component.
Instead, it is a collection of methods, skills,
tools, activities, and ideas. They can be
combined and used in many different ways
to assess the relative value of an investment
over time. Applying this collection in a
particular situation requires making many
choices among the ideas and methods
available and conducting an analysis
appropriate to the decision at hand.
Different choices will produce different
results. Therefore the Guide presents a
framework of the key questions that should
lead to an appropriate ROI analysis. It then
presents a review of the methods and
resources needed along with examples
of different approaches in detailed case
studies.
How extensive should your
ROI analysis be?
Once the decision has been made to conduct
an ROI analysis, what should it look like?
The choice of how to conduct the analysis
should be based on four critical principles
pertaining to:
I
the strategic objective(s) of the ROI
analysis,
I
the place (and importance) of the IT
investment in the overall enterprise
architecture,
I
the type of analysis that should be
conducted (i.e., what data and methods
of analysis are best suited to those
objectives), and
I
how the ROI analysis fits in the overall
decision context for IT investments.
Understanding the strategic
objectives of your ROI analysis
Understanding the strategic objectives of
an ROI analysis will determine how the
analysis is ultimately done and used. A
handful of questions like these can help
managers decide what the objectives of
the analysis should be.
I
Is the proposed project critical to the
business objectives of your agencies or
government?
I
What are the risk factors associated with the
investment?
I
Who will be impacted—positively or
negatively—by the proposed project?
I
Is an ROI necessary for approval and
support of the proposed project?
I
Is the proposed project worth the
investment of an ROI analysis? And if
so, how detailed should it be?
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
1
Answering these questions will help identify
the resources needed to conduct an ROI
analysis, which in some cases can itself be
a substantial investment. Extending its level
of detail beyond what is needed for effective
decision making is a waste of resources.
Focusing on strategic objectives keeps
attention on the full range of benefits to be
expected from the investment, and how to
measure them.
Understanding the context
of your IT investment
Any IT investment project is embedded in an
organization’s technology infrastructure
(enterprise architecture), relevant business
processes, organizational environment, and
external relationships.
2
I
Technology infrastructure. There are
direct costs associated with the technology
and services in which you invest, and
there will also be costs in terms of its
impact on other technology systems
already in place. The benefits range from
more efficient automation and workflow to
improved collection, storage, and access
to information.
I
Business processes. An ROI analysis
must not only account for the improvements to relevant business processes,
but also for the costs associated with
training staff involved in using the
proposed technology system.
I
Organizational environment. Other
costs and returns will be linked to the
organization, for example through altered
resource flows, performance changes,
changes in work flows and internal
relationships.
I
External relationships. Linkages with the
external environment may be significant
as well. Resources may be committed
from this environment to support the
project and additional costs may
be imposed on external persons or
organizations by changes in the way
services are delivered or other business
is conducted.
Choosing the right type of
analysis
Choosing and using the various methods of
ROI analysis requires sound knowledge and
judgment: knowledge about the methods
and judgment about how best to apply
them. The methods chosen should fit the
particular questions asked of an ROI
analysis. Different questions require
different measurement approaches to fit
them. In general, there are four types of
questions that prompt or drive an ROI
analysis: financial, effectiveness, efficiency,
and impact.
Financial: Can we afford this? Will it pay
for itself?
An ROI analysis that answers these
questions is based on expected savings and
revenue increases compared to the dollar
cost of all expenditures on the new system.
The measures are set by generally accepted
or legally mandated accounting standards
and practices that apply to the particular
government organization. The costs and
savings or revenue might be projected over
a multi-year time span to show a payback
period or to estimate the present value of
future returns.
Effectiveness: How much “bang for the
buck” will we get out of this project?
The ROI analysis that will answer these
types of questions considers how much the
investment contributes to achieving program
goals and producing the desired results. It
considers direct, indirect, and opportunity
costs. The indirect costs include such things
as training and administration over time. An
opportunity cost could be the loss of return
or revenue you would have received had
you chosen a different alternative. The
measurement of returns will be expanded
beyond cost savings to include levels of
performance relative to program or project
goals.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Efficiency: Is this the most we can get for
this much investment?
The ROI that tackles this question requires
information about the greatest possible
value relative to its costs. Efficiency cannot
be separated from effectiveness. It is usually
expressed in terms of optimizing the
value of a return for a given cost or input.
Establishing that some particular result is
the best of all possible results requires
either examining many alternatives or
simulating performance in some way that
provides a valid picture of what is possible.
This can be done for some (but not all) kinds
of systems with sufficient resources and
data. However to do so can substantially
increase the cost and complexity of the
analysis.
Impact: Will the benefits to society (our
state, our city, etc.) justify the overall
investment in this project?
The analysis that answers impact questions
will be concerned with the larger social and
economic benefits and costs of a project. To
define and measure variables that represent
social costs or benefits requires more
than the typical economic or accounting
frameworks. These measures are based on
either the specific program results desired
by an agency or on general social benefits
and improved quality of life. Though not
impossible, the breadth and complexity of
this kind of ROI analysis is rarely found in
IT investment planning.
How does the ROI analysis
fit into the overall decision
context for IT investments?
Investment decisions in the public sector,
whether they involve IT or not, necessarily
take place in a context of political and
policy influences. No matter how solid
or technically sophisticated an ROI
analysis may be, it will not likely be the
sole determinant of an investment decision.
When deciding how to prepare and present
an ROI analysis, therefore, it is best to take
into account all the potential risks that
influence the decision process. Undertaking
an ROI analysis should include attention to
the risk factors identified below.
Risk factors that can impact
investment decision process
Politics and policy factors
I
I
I
I
I
Public exposure to failure
Divided authority over decisions
Multiple stakeholders
Year-to-year budget cycles
Highly regulated procurement
processes
Organizational factors
I
I
I
Complex program networks
Misalignment of (or conflicting)
internal goals
Lack of leadership support
Business process factors
I
I
Impact on existing process
Fear of changing work assignments
Technology factors
I
I
I
Rapid changes in technology
Interacting with parallel systems
Scale and complexity
Most of the risk assessment issues listed
here involve problems related to thinking
beyond the boundaries of the project,
measuring factors, or determining
probabilities. Simply recognizing where
uncertainty and potential damage may lie
is half the battle. Careful risk analysis,
based on the best available data and
estimates, will surely assist in ROI analysis
and improve planning, even if the amount
or quality of data is less than ideal.
Considering these various risk factors
can help shape the style, emphasis or
presentation strategies employed to
introduce the analysis into the decision
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
3
making process. Such considerations as
those listed here may also help in recruiting
support for the conclusions of your ROI
analysis and guiding how the analysis
process is positioned when seeking that
support.
Conclusion
There is no single “right” way to conduct a
return on investment analysis. Nor is there a
Consumer Reports for ROI products and
services. In determining how to conduct your
analysis, the best advice is to focus on the
strategic objectives of the analysis along
with the goals and business processes of
the proposed project. This focus will help
guide decisions about the resources and
methods to use to conduct a sound and
valuable ROI analysis.
4
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Chapter One: ROI and the Need
for Smart IT Investment Decisions
“It [ROI] will let us begin to make assessments and decisions about funding a
project or developing a new service based upon some true data. That moves you
from having emotional debates about projects to having factual discussions.”
Gerry Wethington, Missouri CIO1
G
overnment decision makers must make
the most of scarce resources and at
the same time respond to ever-increasing
demands for improved performance and
new technology. Thus the need for wise
investment in information technology
continues to grow. Growing demand in the
face of scarce resources generates hard
questions and close scrutiny of proposals
for new investments. What’s more, the
dismal failure record of many government
IT investments raises legitimate concerns
about the value of these investments
and why they so often fail to live up to
expectations, or even to work at all. As a
result, IT planning processes often include,
or even require, a rigorous business case
to justify new IT investments. These include
ways of assessing the costs and returns to
be expected from that investment, that is,
return on investment (ROI) analysis. This
Guide is designed to help government
executives who need to design, direct,
conduct, or work with the results of such
an analysis to make the most of their
investments in information technology.
Growing interest in assessing returns on IT
investments has spawned wide interest in
methods of return on investment analysis.
However, there is little agreement about
best practices or specific methods for
assessing ROI. Professional publications
and consultant white papers present quite
a variety of possible approaches. As a
result, government executives and decision
makers have difficulty choosing or designing
a return on investment analysis that is both
feasible and appropriate to their needs. To
help administrators and decision makers
1
with these choices this guide presents an
overview of the purposes and concepts of
ROI along with an introduction to basic
methods and example cases. It also
includes links to other resources for those
who wish to explore some subjects in
greater depth.
This guide treats ROI analysis as part of
the overall decision making process for IT
investment. The planners and designers of
an IT project can use ROI analysis to help
persuade decision makers to support the
project. Decision makers can use an ROI
analysis, indeed may even require one,
as part of an IT investment proposal to aid
them in evaluating it. In either case, an ROI
analysis will be shaped by the situation in
which it is designed and carried out.
Decisions about what sort of return on
investment analysis to do, or whether to
do one at all, will usually depend on a
variety of factors. ROI analysis may or may
not fit the larger context of investment
decision making. Decision situations
driven by very short deadlines or highly
specific policy directives may rule out
extensive analysis. Scale matters as well.
An elaborate ROI analysis would hardly be
justified for a small-scale, low-risk project
that requires a fast decision. By contrast,
large complex projects are typically highrisk propositions for which the added time
and cost of an extensive ROI analysis would
be fully justified. Even though justified, in
some environments ROI analysis is not
used at all in favor of best practice reviews
or benchmarking to evaluate investment
possibilities. Current practices vary
“Running the Numbers,” Government Technology, June 2002, p. 23.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
5
considerably. In Iowa, for example, the
state government has a standard framework
for all agencies submitting IT project
proposals, including business case and
ROI analysis requirements.2 The US Office
of Management and Budget has imposed
similar requirements on federal agencies.
Agencies and local governments in other
states may develop their own internal
business case and ROI requirements.
Given the diversity of practice in the IT
investment world, this guide takes an
eclectic, non-prescriptive approach. It
treats the subject of ROI analysis from the
point of view of a curious but uncertain
decision maker. The key issues facing this
decision maker are whether to do an ROI
analysis and, if so, how. The guide does
not advocate engaging in ROI analysis
under all circumstances, nor does it favor
any particular technique. Instead it presents
an approach to understanding a range of
purposes and methods for ROI analysis
that can assist that decision maker to
move forward with wise and effective IT
investment choices.
An approach to
understanding and
using ROI analysis
Decisions about how to use ROI analysis
depend on understanding the nature of
the methods themselves and how they
relate to the business setting. ROI analysis
in general is a rather diverse collection of
methods, skills, tools, activities, and ideas.
They all may be useful for assessing the
relative value over time of some investment.
These methods are not, however, a single
formula or predetermined calculation that
will yield a simple yes-or-no answer to
the question of how to invest. ROI is not
a silver bullet. Actually designing and
carrying out any kind of ROI analysis
requires making many choices among
the ideas and methods available and
conducting an analysis appropriate to the
decision situation. Different choices will
produce different results.
Consequently, a meaningful analysis
of returns on investment in information
technology is far easier said than done.
Choices about how to conduct an ROI
analysis should be based on critical
understandings about:
I
the strategic objective(s) of the
analysis,
I
the place of the proposed IT
investment in the overall enterprise 3,
I
exactly how the analysis should be
done (i.e., what data and methods
of analysis are best suited to those
objectives), and
I
how the ROI analysis fits in the overall
decision context for IT investments.
This guide will introduce you to these
four basic understandings and provide
resources for deeper investigation of each.
2
The Iowa ROI approach can be found at http://www2.info.state.ia.us/roi/index.html. The Federal business case requirements are found in OMB
Circular No. A-11.
3
The enterprise can be a single agency or unit, or something as broad as the education or justice enterprise. The U.S. Chief Information Officers
Council defines an enterprise in terms of enterprise architectures, or “blueprints” for systematically and completely defining organizations’ current
(baseline) or desired (target) environments. See A Practical Guide to Federal Enterprise Architecture, Version 1.1, Chief Information Officers
Council, February 2001, p. 2. More specifically, the National Association of State Chief Information Officers defines enterprise architecture as an
overall plan for designing, implementing and maintaining the infrastructure to support the enterprise’s business functions and underlying networks
and systems. See Enterprise Architecture Development Tool-Kit, Version 2.0, National Association of State Chief Information Officers, July 2002,
p. 243.
6
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Understanding strategic
objectives in ROI design
Your understanding of the strategic
objectives of an ROI analysis will determine
how the analysis is ultimately done and
used. The matter of strategic objectives has
two related parts. One deals with the
objectives and context of the proposed
investment. The second deals with the
objectives and context of the ROI work itself.
An adequate understanding of the first part
of the strategic objective must include
answers to these key questions detailed
below.
What are the programmatic and business
goals of the proposed investment?
The value of an investment is directly
related to the programmatic goals and
business process employing the new
technology. For example, the strategic goal
of investing in a new financial management
system would be to improve the quality
of financial decision making and control of
resource flows, not simply to produce faster
or more complex accounting reports.
Attention to the strategic objectives keeps
attention focused broadly on the kinds of
benefits sought and how to measure them,
rather than narrowly on the technology.
Focus on the strategic and business
objectives will also lead to a clearer
understanding of the full range of benefits
to be expected from the investment.
What are the needs or interests of the
primary customers and other key
stakeholders with respect to products
or services involved and impacts on
business processes?
Stakeholders are critical players in the
success of new projects. Ignoring or
underestimating the importance of their
role puts the success of an investment
at great risk and can lead to substantial
unanticipated costs or missed benefits.
One large organization recently developed
an extensive new customer help desk
application without the participation of the
staff who handled customer phone calls.
Those staff members learned of the new
system the day before it went live. The result
was poor performance and a serious blow to
staff morale—lower returns and higher
costs.
How extensive is the analysis to be? What
range of costs and returns are expected?
What resources and frameworks are
available or required for the ROI analysis
itself?
An ROI analysis can itself be a substantial
investment. Extending its scope, level
of detail, or complexity beyond what is
needed for effective decision making is
a waste of resources. Attempting an ROI
analysis that is beyond the resources or
capacity of the organization will also waste
resources. Choosing an appropriate scope
for the analysis is therefore a critical part
of the process. These choices determine
the details of the ROI analysis itself: kinds
of data used, whether hard or soft estimates
will suffice, the kinds of projections,
quantitative or qualitative data that are
needed; which financial or non-financial
outcomes (customer/user satisfaction,
social or political outcomes, improved
equity) are all important. The answers to
these questions then influence the kinds
of personnel, tools, and other resources
needed.
What are the risk factors and how
might they affect the project’s costs
and results?
Consideration of risk should always be
a part of the ROI analysis. Risk factors
arise from the nature of the project
itself (complexity, scale, novelty), its
organizational setting (conflict, resource
constraints, top-level support, time
pressures), and the larger environment
(political turbulence, crises, and policy
shifts). Risk analysis, discussed in some
detail later in this chapter, examines the
likelihood of risk factors affecting the
project and what elements of the project
or its results are likely to be affected.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
7
Table 1. IT ROI Questions from the CFO to the CTO4
REQUIRED INVESTMENT
How much investment–including capital
expense, planning and deployment,
application development, and ongoing
management and support–will the project
require?
FINANCIAL BENEFITS
What are the expected financial benefits
of the project, measured according to
established financial metrics, including
ROI, … savings, and payback period?
STRATEGIC ADVANTAGE
What are the project’s specific business
benefits, such as operational savings,
increased availability, increased …
revenue, or achievement of specific …
goals?
Defining and measuring
the costs and returns from
IT investments
If an ROI analysis were just one simple
thing, then there would be one simple way
to measure the costs, returns, and benefits.
In practice, however, there can be many
different questions asked of an ROI
analysis requiring different measurement
approaches to fit those questions. How
to identify these measures and apply them
to analysis are the next parts of the puzzle.
Solving that puzzle requires understanding
the differences among the questions asked
of an ROI analysis. There are four different
but related types: financial, effectiveness,
efficiency, and impact questions.
4
8
IT OPERATING EFFICIENCY
How will the project improve IT, such
as simplifying management, reducing
support costs, boosting security, or
increasing IT productivity?
RISK
What are the potential risks associated
with the project? How likely will risks
impact the implementation schedule,
proposed spending, or derived target
benefits?
COMPETITIVE IMPACT
How does the proposed project compare
with competitor’s spending plans?
ACCOUNTABILITY
How will we know when the project is
a success? How will the success be
measured (metrics and time frames)?
Can we afford this? and Will it pay for
itself?
Answering these questions requires
information about costs and returns in terms
of the monetary value of the resources used
(inputs), as measured and recorded by
standard financial factors. In its simplest
form, an ROI analysis based on this kind
of question would calculate the return in
terms of the expected savings and revenue
increases (if any) compared to the dollar
cost of all expenditures on the new system.
The costs, savings, and revenues might be
projected over a multi-year time span in
order to show a payback period or to
estimate the present value of future returns.
Tom Pisello, in Infoworld, June 10, 2002, p. 47.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
How much ‘bang for the buck’ will we get
out of this project?
Answering effectiveness questions requires
information about the costs of the project in
relation to how much it contributes to
achieving program goals and producing
the desired results. The metrics will be
more complex, involving unit cost or activity
cost calculations. The measurement of
returns will be expanded beyond cost
savings or revenue increases to include
levels of performance relative to program
or project goals.
Is this the most I can get for this much
investment?
Answering efficiency questions requires
information about whether the project will
produce the greatest possible value relative
to its costs. Efficiency questions pose
serious analysis problems. Establishing
that a particular result is the best of
all possible results requires either
examining many alternatives or simulating
performance in some way that gives a
valid picture of what is possible. This can
be done for some kinds of systems with
sufficient resources and data, but can
substantially increase the cost and
complexity of the analysis.
Will the benefits to society (our state or
our city) justify the overall investment in
this project?
Answering impact questions requires
information about the larger social and
economic benefits and costs of a project.
These questions pose two tough problems.
Measuring the broad social and economic
costs of an investment requires data far
beyond what typical financial systems
provide. Measuring, or even identifying
the full range of social and economic
benefits from some government IT project
can be even more daunting. The idea of
figuring a cost/benefit ratio is appealing,
but seldom feasible. Though not impossible,
the breadth and complexity of this kind of
analysis means it is rarely found in IT
investment planning.
The four types of measurement questions
and approaches differ in several ways.
The choice of relevant metrics is one critical
difference. Some approaches are based on
strictly financial metrics (costs or returns in
dollar terms), others include production
output metrics, such as quality of goods
or services, client or customer satisfaction.
Metrics may extend to organizational factors
such as morale or to social and political
outcomes. These can include impacts on
quality of life, social equity, social or human
capital, or political support. These
differences are summarized in Table 2
(page 11).
Understanding the
enterprise: technology in
the business context
This guide views any IT investment project
as embedded in its organization’s
enterprise architecture and in a context
with three major elements: relevant
business processes, the organizational
setting, and the external environment. 5
These are illustrated in Figure 1 on page 10.
The immediate context of any IT project is
found in the current business process(es).
Some of the costs and returns of the project
will be directly linked to business processes,
such as training costs for staff involved or
the improved efficiency of the overall
process resulting from project implementation. Other costs and returns will be linked to
the organization, for example through
5
While a more detailed discussion of enterprise architecture is beyond the scope of this guide, both the mapping and understanding of an
organization’s “blueprint” are critical steps that any organization should accomplish prior to conducting effective return on investment analysis.
Please see the Additional Resources section in Appendix D for several links to enterprise architecture development tools available to federal,
state, and local government decision makers and ongoing enterprise architecture initiatives at the state level.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
9
changes in resource flows, performance
changes and changes in workflow and
internal relationships. Linkages with the
external environment may be significant
as well. Resources may be committed from
that environment to support the project,
and additional costs may be imposed on
external persons or organizations by
changes in the way services are delivered
or other business is conducted.
For example, an agency implementing EDI
(electronic data interchange) to support
purchasing transactions may be imposing
costs on vendors who wish to do business
with the agency and have to invest in
developing their own EDI capabilities. The
way Figure 1 below represents investment
costs and returns as part of the same
context is an important way of looking at
ROI. Neither the costs nor the benefits of an
IT project begin and end at the project’s
boundaries. Financial allocations to a new
project mean fewer resources somewhere
else in the organization or its environment.
Changes in one part of a business process
may impose costs on other parts that have
to adjust activity, retrain staff, or modify other
systems. Increased efficiencies in one
business process can make resources
available elsewhere in the organization,
but may also result in changes in other
linked processes within the enterprise.
Whether you see something as a cost or
benefit of an IT project may depend on
whose perspective you take. The expenditure on a new personnel management
application is a cost to the agency that pays
for it, but it is a benefit to the vendor, who is
an external stakeholder. These links make it
clear that an analysis of costs and benefits
of an investment can require attention
extending well outside the project itself
into the organization and its environment.
Business processes are the critical
connection between the project and the rest
of the organization. So attention to business
process linkages with the project is an
important part of the overall ROI analysis.
Figure 1. The Context of IT Investment Projects
10
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Table 2. Approaches to Cost and Return Measurement
Measurement
Question
Measuring
Costs
Measuring
Returns/Benefits
Can we afford this
and will it pay for
itself ?
Financial metrics; defined by
policy and accepted accounting
principles ; reporting and
control-oriented; standardsbased or consistent; not linked
to business process; ignores
important cost factors; short
time frame; data routinely
collected/repor ted
Savings as measured in
accounting categories; narrow
in focus and impact; increased
revenues, reduced total costs,
acceptable payback period
How much ‘bang for
the buck’ will we get
out of this project?
Financial and outcome/quality
metrics; operations and
management oriented; defined
by program and business
process; may or may not be
standardized; often requires
new data collection; may
include organizational and
managerial factors
Possible efficiency increases;
increased output; enhanced
service/product quality;
enhanced access and equity;
increased customer/client
satisfaction; increased
organizational capability;
spillovers to other programs
or processes
Is this the most I can
get for this much
investment?
Financial and organizational
metrics; management and
policy oriented; non-standardized; requires new data
collection and simulation or
analytical model; can reveal
hidden costs
Efficiency increases;
spillovers; enhanced
capabilities; avoidance of
wasteful or suboptimal
strategies
Will the benefits to
society (our state,
our city, etc.) justify
the overall investment in this project?
Financial, organizational,
social, individual metrics;
individual and management
oriented; nonstandard; requires
new data collection and
expanded methods; reveals
hidden costs; potentially long
time-frame
Enhanced capabilities and
opportunities; avoiding
unintended consequences;
enhanced equity; improved
quality of life; enhanced
equity; enhanced political
support
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
11
Understanding ROI
decisions in their political
and policy context
Investment decisions in the public sector,
whether they involve IT or any other
expenditure, take place in a context of
political and policy influences. No matter
how solid or technically sophisticated an
ROI analysis may be, it will not likely be the
sole determinant of an investment decision.
It is useful in deciding how to prepare and
present an ROI analysis, therefore, to
take into account possible political and
organizational factors. Such a consideration
of external factors may help shape the style,
emphasis, or presentation strategies
employed to introduce ROI analysis into
decision processes. Such considerations
are discussed below and may also help in
recruiting support for the conclusions of the
analysis and guiding how the analysis
process is described or defended.6 All of
these considerations can be classified as
different types of risks.
Political risk factors
Public exposure of failure or error.
Government’s business is public business.
Most new ideas are implemented in full
public view. Any investment-gone-wrong
risks not only dollars, but the credibility
of an agency and its leadership with
legislators, executive officials, and the
public. So government tends to reduce risks
by relying on the “tried and true.” Failure
risk can be mitigated by taking care not to
over promise the benefits of new projects
and to ensure that there is adequate
strategic planning to reduce the probability
of failure. Undue caution can also risk
a different kind of failure: missed
opportunities for successful projects.
Divided authority. Public executives
seldom have a clear line of authority over
agency operations. Their decisions are
circumscribed by existing laws, budget and
financial controls, civil service systems,
political constraints, and a variety of
regulations imposed by both legislatures
and the courts. These restrictions impede
managing the complexities of multi-million
dollar IT projects in a rapidly changing
technical environment.
Multiple stakeholders. Stakeholders
typically have competing goals. Customers,
constituents, vendors, service providers,
elected officials, professional staff, and
others all have some stake in IT projects.
Understanding how different choices may
affect and be affected by each stakeholder
group helps to prevent unexpected
problems.
Annual budgets. Government budget cycles
increase the uncertainty about the size
and availability of future resources. This
diminishes government agencies’ abilities
to adopt or sustain new IT innovations
successfully, especially those that have long
development periods.
Highly regulated procurement. Regulations
in the typical competitive bidding process
are ill suited for the experimentation and
learning that is often essential for successful
IT investments. While promoting integrity
and fairness, procurement regulations are
often a source of problems and delays.
These are especially troublesome when
agencies write requests for proposals
(RFP’s) that depend on the limited
information they have been able to gain
from inadequate experience and research.
6
The content of this section is adapted from Sharon S. Dawes, Theresa A. Pardo, Stephanie Simon, Anthony M. Cresswell, Mark F. LaVigne,
David F. Andersen, and Peter A. Bloniarz. Making Smart IT Choices: Understanding Value and Risk in Government IT Investments. Albany, NY:
Center for Technology in Government, 2003. http://www.ctg.albany.edu/publications/guides/smartit2
12
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Organizational risk factors
Technology risk factors
Complex program networks. Government
programs are connected in many complex
ways to other programs in the same or other
agencies, or to non-governmental entities.
Sometimes the connections are explicit
and formal. Often they are informal or
unintended. Changing programs can have
unintended consequences for several
others, producing additional costs and
problems for the investment project.
Rapid change. Obsolescence is a risk as
soon as a project chooses its technology.
The pace of technological change makes
it difficult for planners to keep up with the
details of new developments and to
understand comprehensively how each
new technical tool works, or may be
superceded. Technology choices guided
by long term strategies and strong
linkages to business goals can mitigate
many of the risks produced by rapidly
changing technology.
Misalignment of goals. Some parts of
an organization may see the goal of an IT
project in narrower, possibly conflicting
terms. For example, an IT unit may become
enamored with a database or office
automation project because of its technical
elegance. The end users of the system,
however, may want capabilities that are not
compatible with the new technology. Without
some goal alignment a project is on the path
to failure.
Lack of leadership support and
organizational acceptance. Top
management support for a technology
initiative is critically important. Similarly
support and acceptance throughout the
organization, especially among the people
who will use the technology or its products,
are equally important, and often more
difficult to achieve. Understanding and
enhancing support reduces or limits risks.
Technology interactions. A basic concept
of system analysis is “you can’t do just one
thing.” New technologies interact with old
technologies and work processes. The
interactions may enhance the value of the
older and newer technology, or interfere with
both. A careful analysis of these interactions
will identify risky situations and provide
insights for avoiding problems and errors.
Scale and complexity. Risks increase
directly with the scale and complexity of IT
projects. Planning an incremental process
of development, with careful contingency
plans, can mitigate some of these risks
and will avoid problems that cannot be
anticipated on the path from small to large,
complex systems.
Business process risks
Reality of the process. IT inventions are
embedded in business processes. Failure
to understand and account for this reality
in project design and implementation
introduces major risks. Systems are often
created that do not serve business needs,
are too expensive for the small productivity
gains they provide, or are not flexible
enough to meet changing demands.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
13
Chapter Two: Methods of ROI
Analysis for IT
C
hoosing and using the various methods
of ROI analysis requires both good
knowledge and good judgment: knowledge
about the methods and judgment about
how best to apply them to a particular IT
project and its setting. This section presents
an introduction to the methods. It is intended
to impart enough knowledge so that the
reader can exercise effective judgment
about whether to use a particular method,
how and to what extent it may be applied,
and where to go for more complete
information and resources.
We begin with a discussion of the issues
associated with project time frames and
scale, then present various approaches to
process analysis and measurement. We
conclude with a brief overview of risk
analysis.
Issues of time and scale
The selection of method will depend to
some degree on the time frame and
the scale of the project and the ROI
analysis. Time frame refers to both the
time perspective for the analysis and the
overall time period of the project to be
included in the analysis framework.
Anticipating the future. ROI analysis is
commonly used prospectively. The results
of the analysis are intended to inform a
decision about a future IT investment.
For a prospective analysis, estimates of
anticipated cost and performance are based
on assumptions about the future that may
involve considerable uncertainty. The
results of the analysis will depend, to a
large degree, on how accurate those
assumptions turn out to be.
Learning from the past. A retrospective ROI
analysis will show actual performance data
about the IT project’s costs and returns. The
longer timeline and complexity of larger
projects can lead to substantially different
14
results. The analyst must take these
possibilities into account when evaluating
the results of pilot projects. These concerns
become part of the risk analysis discussed
in a later section.
Importance of scale. Even when there are
actual data from such a pilot project, the
problem of the accuracy of assumptions
remains. There is no guarantee that costs
or returns from a small pilot project will
accurately predict what happens in a
larger effort. The scale of the larger effort
can itself lead to different results.
In the case study described in Appendix A
(page 28), the analysis combines the two
perspectives. The developers created a
pilot project, which was a small part of a
larger project they planned to undertake.
They then used the data gathered from a
retrospective analysis of the pilot project
to estimate the costs and returns that would
result from the full development.
Importance of business
process analysis
IT projects can have large and wide-ranging
impacts on business processes. These
impacts can occur directly in the business
processes involved in the project and in
other related processes. Methods for the
analysis of business processes are too
large a subject to cover in detail in this
guide, so a brief introduction to these
techniques is included in the section on
methods below. The reasons for including
attention to these methods in ROI work is
related to the overall strategic objective of
the project and how it fits into the rest of the
organization.
The business process perspective concentrates on where an IT project fits within the
larger picture of the organization’s mission
or core purpose. For example, a state
department of transportation typically
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
has responsibility for ensuring the safety
of the transportation infrastructure. This
would usually include responsibility for
inspecting bridges on state highway
systems and contracting for repairs where
needed. The main elements of that business
process includes many activities, from
decision making about inspection policies
and schedules through contracting for
repair and construction work. The activities
could include scheduling and performing
inspections, reporting and analyzing
results, then moving into the RFI, RFP
development, bidding, contracting, and
project supervision. Any change in one
part of the IT system in an interconnected
business process like this one, would likely
have important impacts on other business
processes. An IT project that places
electronic sensors on bridges that send
stress data to the central office will affect
more than the inspection process. It will
have ripple effects all the way from the
employment and training of inspectors to the
costs of new construction and the public’s
perception of bridge safety. These latter
concerns are all potential elements in an
ROI analysis.
The interconnectedness of IT projects and
the overall business process is nicely
illustrated in one recent experience with
a Web project in New York City. The City
government added a section on its Web
site reporting the results of its Health
Department inspections of restaurants.
The IT staff did not anticipate the popularity
and high demand for such information. Nor
did they understand the effects that demand
would have on other linked elements of the
business process, which includes both
producing and disseminating inspection
results. The Web server was initially
overwhelmed by hits on the inspection
information soon after it was posted; word
of mouth communication rapidly spread the
news. That problem could be easily fixed
with additional server capacity. However,
the high-level of public demand for this
information and its significant impacts on
restaurateurs put new pressures on the
Health Department. They were pressed
to increase the timelines of inspections
and to use more plain language to ensure
better understanding of the results. These
were substantial cost factors directly related
to the Web site project. A narrow focus on
the IT component of the project led the
planners to miss the effects on linked
elements of the business process. The
results were both unanticipated costs
resulting from down-stream impacts on
the business process and unanticipated
benefits due to increased service levels to
the citizens.
The linkages in this example extend from
the specific details of an IT project—new
data on a Web site—all the way to the
relationship of the organization to its
political environment. These linkages
and their importance are obvious in
retrospect, as are many implications for
both costs and returns. The goal of good
business process and ROI analyses is to
provide the same insights and information
for planning and decision making in
advance. How that can be done is outlined
in the description of methods below.
Benefits of process modeling
Business process models make the implicit
assumptions and mental models of
individual managers and stakeholders
more explicit and open for discussion.
They also:
I
inhibit premature jumping to a solution
because of the way they structure
thinking about a problem;
I
create an externalized definition of the
problem that can serve as a focus of
discussion and can help to align thinking
about what are root causes of observed
problems;
I
force managers and analysts to come to
grips with the precise logic or causal
forces that are causing a problem;
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
15
I
require attention to decide which key
variables to measure;
I
allow analysts and managers to
communicate their reasoning effectively
and efficiently to external audiences;
I
can provide a simulation of how the full
system will operate within a full context
of organizational and human factors;
I
push managers to see the implications
of a limited prototype when it is
expanded to full scale operations,
requiring careful attention to technical,
organizational, and policy issues; and
I
can include financial elements that allow
explicit exploration of costs and benefits
of proposed solutions.
Sensitivity analyses of models provide
answers to “what if ” questions about various
types of system functionalities and possible
organizational and human effects. This
helps planners anticipate issues and
problems before they are encountered
in a real world system implementation.
Methods for understanding
the business process
IT investments of any sort must be viewed
as part of the business processes in their
organizations. Both the costs and returns
of the investment are tied to how the new
technology fits with and how it alters
business activities. The notion of what
constitutes IT must be expanded to include
not only the chips, wires, and software, but
also the activities and interactions that
generate the costs and value that results
from using the IT (i.e., the enterprise
architecture). Failing to map and
understand the enterprise invites a badly
flawed understanding of how the IT
investment will work, and can be a short cut
to failure. For example, the U.S. Department
of Education recently invested millions of
dollars in a Web site for college students to
apply for financial aid. The site generated
very little use and was closed down. It
failed because it ignored the important
16
role of college financial aid officers in the
overall process of students seeking and
receiving aid. The designers did not
understand or ignored the overall business
process of financial aid administration.
Business process analysis tools and
methods range from simple flow and
GANTT charts to sophisticated formal
and mathematical modeling. Simple flow
charts are sufficient to identify the basic
elements of most business processes and
some of the more obvious dependencies.
How far business process modeling needs
to go beyond simple diagrams depends
on the questions to be answered and
the resources available for the work.
The basic components of a business
process are activities, flows, controls,
and dependencies. Analyzing the
business process therefore means
identifying the various kinds of activities,
what kinds of flows they generate and
support, and how the activities and flows
are controlled and linked to produce some
particular collection of goods or services.
Flows can consist of information, persons,
resources, control inputs, and work
products moving from one activity to
another. The analysis consists of gathering
information about the activities and flows
in as much detail as necessary. Depending
on the kinds of modeling to be used, this
information can include descriptions of
activities and flows in strictly qualitative
terms or detailed measurements of
resources used, flow metrics, and outcome
measures.
Descriptive models
Approaches to process modeling can be
grouped roughly into three levels or
types: descriptive, analytical, and
dynamic. Descriptive models are the
most fundamental. They describe the
elements of a business process and the
relationships among them. A flow chart
that describes how travel expense
reimbursements are processed, for example,
might show how the request originates, how
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
it moves from one work process to another,
what actions or decisions take place at each
point, and the elapsed time for each step.
Descriptive models may involve some
measurements, such as personnel time,
expenditures, other resources used, and
outputs. The flow chart in Figure 2 below
illustrates the actual billing information flows
for a nonprofit provider of services to
mentally retarded and developmentally
disabled clients. It shows how information
about services delivered is converted into
bills submitted to the various government
sources of funding for these client
services.7 This flow chart was produced
to help understand how a change in the
reporting and processing technology for
one of the funding agency’s would affect
operations among the providers who deal
with multiple agencies. The IT project
was planned by one state agency, but
the planners recognized that the business
process involved a range of other agencies
and local practices. Therefore their analysis
of the costs and potential benefits of the
project had to include attention to the
overall business process, not just what
happened within one state agency.
Figure 2. Example of Process Flow Chart
7
For example, Medicaid billing goes to the Medicaid Management Information System (MMIS), education related billing goes to the State Education
Department, and so forth.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
17
The level of detail used for such a model is
a matter of judgment, in this case dictating a
model at an intermediate level of detail. For
some purposes, a more highly aggregated
model may be sufficient, such as the
following.
It might be necessary to examine a
much finer level of detail, such as how
administrators review service records
before approving billing in Figure 2 on
page 17. The level of detail depends on
the questions that guide the overall effort.
A descriptive model influences the choice
of what is to be measured and whether
more formal modeling will be needed.
Use of analytical or
formal models
Analytical or formal models go beyond
description to represent the performance
of a business process or project in some
quantitative or mathematical form. This
kind of model has many advantages for
analysts and decision makers, along with
some substantial problems and limitations.
There are at least four kinds of advantages.
I
I
I
18
Formal models require making the
implicit explicit, and to do so in precise
and systematic ways.
Many formal modeling methods allow
exploring “what if ?” questions and work
with alternative scenarios by simulating
the results of using the model of the new
system. The planner can thus test ideas
and alternatives without risk to real
systems, operations, or data.
The analysis yielded by the modeling
can reveal insights into the behavior
of persons or organizations that would
otherwise be difficult or impossible to
detect.
I
The modeling process, if done in a
collaborative way, produces a shared
understanding of the work among
managers and stakeholders that can
lead to improved decisions about the
project and its ultimate implementation.
Because of these advantages, the use
of these techniques for process improvement and project planning has grown
substantially and spawned an industry
producing software applications, consulting,
and training in one or more of these
methods (see resources in Appendix D).
These methods are not without problems
or limitations, the most important of which
is that they raise the costs of analysis in
several ways. These models typically
require more detailed and extensive
information, particularly performance
and outcome measures, than simpler
descriptive methods. In addition, the
construction and use of these modeling
methods requires considerable time and
expertise. Organizations that do not have
staff prepared to do the modeling work
themselves would have to invest either in
extensive training or employing consultants.
It is important to review carefully whether
the complexity of the project justifies the
investment in formal modeling before
taking that course of action. The review
of alternative methods below is intended
to help in that regard.
Types of formal
modeling approaches
Agent-based modeling. The basic idea of
agent-based modeling is that complex
behaviors of collectives (social groups,
members of an organization, population)
can be modeled by simple computational
rules. The rules treat the members of the
collective as autonomous agents that follow
simple rules of behavior, rules that can be
modeled on a computer. What happens to
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
the collective is the result of the individual
agents interacting with each other according
to the rules. Changes in the rules change
the overall results, sometimes in predictable
and sometimes unpredictable ways.
One early example of agent-based
modeling in the social arena was a model
of residential segregation developed in
the late 1960’s.8 The model starts with
a hypothetical city with a mixed race
population randomly distributed over
the residential area. Each household
either stays or moves according to two
simple rules: a preference for living near
persons of the same race, and deciding
to move if the population in surrounding
areas has too low a proportion of such
households. There is no rule involving
dislike for households of different race
or desire to avoid them. Nonetheless,
the action of these rules over time
produces a city of racially segregated
neighborhoods.
In agent-based modeling, the rules are
the model or theory. Thus they make
explicit the assumptions and motives of
the agents and what governs their actions.
The value of the model depends on the
degree to which the rules produce results
that match the actions and outcomes for
the actual collective of interest. If rules
are developed that reproduce the results
of interest, they can provide a powerful
tool for understanding how those results
occurred and for testing ideas about how
to change or improve them. This kind of
modeling may be particularly appropriate
for information technologies that will
involve the activities of large numbers
of persons engaged in the same type of
activity, such as large numbers of citizens
seeking the same kinds of information on
a Web site. A valid model of how people
will use a system can be a valuable tool for
exploring what costs and benefits to expect.
8
Agent-based modeling is not for the
faint of heart, however. It is mathematical,
often requiring advanced knowledge of
algorithms and computing to implement.
There are special modeling languages
for developing the computer programs
to pursue agent-based work, but
they require experienced users or a
considerable investment in training. If the
rules are sufficiently complex or abstract
they may be difficult to communicate to
participants in the planning or to key
stakeholders.
Operations research and statistical
modeling. If the factors that influence a
business process can be identified and
measured, modeling that process by
operations research and statistical
methods may be feasible. These modeling
approaches use mathematical equations to
represent the business process. This
requires measurements of the business
process itself and of the factors that are
thought to influence how the process works.
Then the modeling equations are chosen to
fit the way the significant elements of the
business process are conceptualized.
I
If the process is seen as a series of
activities that consume resources
and occur in particular interdependent
sequences, project description tools
such as PERT charts or critical path
analysis may be used.
I
If the sequence of events in the
business process is uncertain, or
if multiple paths or outcomes are
possible, probabilistic methods such as
Markov chain models may effectively
represent the process.
I
If the sequence or path of influence
is not known, but some measure
of their general effects is available,
linear modeling and regression methods
may be useful to represent the overall
process or predict the results of changes.
Thomas Schelling, (1971) “Dynamic Models of Segregation,” Journal of Mathematical Sociology 1, 143-186.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
19
I
If particular combinations of influences
are thought to work together to affect
the flow and/or performance of the
business process, then scenario
analysis may be useful.
These modeling methods are similar to
agent-based methods in that they require
measurements. They differ in that they
are based less on specific theories or
assumptions about the activities that
make up the business process. Rather,
they use simplifying assumptions
about the underlying cause and effect
relationships that drive the process. That
makes the models easier to use in some
respects. Little original theory building is
needed. In addition, the methods are
supported by commonly available
applications, and training in many of the
particular methods is typically part of
management education programs.
Choosing among these alternatives is not
a straightforward task and a reasonably
high level of expertise is needed. The
same applies to interpreting the results
and understanding the limiting assumptions
that apply to each.
System dynamics modeling. Many
business processes involve feedback,
that is, chains of effects in which what
happens at one point in the process has
an impact elsewhere that in turn influences
(increases or decreases) the original effect.
For example, a new database application
may produce a new kind of report that users
find particularly valuable. Seeing one
possible new outcome prompts the users
to request additional kinds of reports that
produce more valuable results, and more
new requests, and so forth. Any kind of
process that can be represented as stocks
or supplies that flow from one place to
another with either reinforcing or negative
feedback can be represented using the
methods of system dynamics modeling.
These models can provide both a conceptual and mathematical representation of the
processes. The conceptual or graphical
representation is usually referred to as a
20
causal loop model. These models can be
built by analysts working with the staff
involved in the process, making use of staff
knowledge and insights. The image created
is a potentially powerful tool to evoke
knowledge about the process and to
move the participants to a shared
understanding. An example of such a
diagram is shown in Figure 3 (page 21).
This particular model represents a working
theory about how work progresses. Tasks
can exist in one of the four following states.
I
Work to do
I
Undiscovered rework
I
Known rework
I
Work really done
At the start of work, all tasks are in the
stock of work to do. As participants perform
work, correctly done tasks move to the
stock of work really done, with a probability
of 1- error rate (a fraction). The error rate
is based on the idea that it is impossible
to perform all tasks correctly on the first
try. Tasks performed incorrectly require
rework, so as work proceeds some proportion of the tasks enter the accumulation
undiscovered rework, according to the
probability of an error occurring, error
rate. Similarly, problematic tasks can
be redone correctly (moving from the
accumulation of known rework to work
really done) or incorrectly (returning to
undiscovered rework), based on the error
fraction. The concreteness of instructions
and transformability of the tasks themselves
are shown as affecting the ability to do
work right in the first place or to correct
errors. With appropriate assumptions
and values for the variables and
parameters in such a model, a mathematical
representation can be built that reproduces
the behavior of the process itself. Analysts,
who should have appropriate expertise in
system dynamics modeling, can change
assumptions or values and explore the
consequences, yielding new insights into
the possible costs and performance of a
project.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Figure 3. Causal Loop Diagram of Project Management Processes
Unified Modeling Language (UML) and Use
Case Modeling. UML is a programming
and modeling language that provides a
system for representing and documenting
object-oriented software development and
the business process in which it resides.9
It is particularly tailored to analyze the
application development process and
for use by development teams in that
environment. The overall conceptual
scheme for use of the UML includes what
is referred to as Use Case Modeling as a
component. A Use Case is a guide for
software design based on a behavioral case
study of how an application is used in an
actual business process. This differs from
traditional approaches that specify software
requirements in terms of technical needs
and goals. For IT projects that are primarily
application development, UML can be a
very useful and powerful tool for modeling
the project itself for management purposes,
as well as for its linkages to the business
process. However, the UML is highly
technical and requires considerable study
and programming experience to use
effectively.
Investment in the use of UML may be
justified, especially for large application
development projects that involve multiple
teams. As the UML documentation states:
“In the face of increasingly complex
systems, visualization and modeling
become essential. The UML is a
well-defined and widely accepted
response to that need. It is the visual
modeling language of choice for building
object–oriented and component-based
systems.10”
9
“The Unified Modeling Language (UML) provides system architects working on object analysis and design with one consistent language for
specifying, visualizing, constructing, and documenting the artifacts of software systems, as well as for business modeling.” Object Management
Group, p. xi.
10
Object Management Group. OMG Unified Modeling Language Specification, Version 1.3. Framingham, MA: Object Management Group
Headquarters, 1999, p. 1-4. 13 Office of the New York State Comptroller. Accounting and Reporting Manual. Albany, NY (no date), p. 17.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
21
Workflow modeling. The workflow
modeling approach is similar to UML but
involves a different focus and approach.
Instead of using a programming language
to model work processes, workflow methods
use generic models of work processes
and the relationships among them. The
components of these generic models
represent business activities, resources,
dependencies, and controls. The design
of the models is aimed particularly at
enhancing coordination by identifying
dependencies, failure modes, and
problems of handling exceptions. For
complex business processes, a flow chart
will not reveal the consequences of errors
at critical points, unanticipated resource
shortages, or missing factors during some
point in the process. Workflow models allow
these situations to be explored and provide
analysis of where processes are particularly
vulnerable to failure or waste.
Workflow modeling technologies and
commercial products may be useful to
analyze an existing business process
or re-engineer one. But they are designed
primarily to develop automated work
processes. These depend on computerbased controls, typically where the flow of
information or materials involves networked
links among the activities. However, the
basic concepts of dependencies, exception
handling, and error modes may be applied
regardless of the level of automation
in a workflow. So the conceptual tools
of workflow analysis may contribute to a
process model concerned with costs and
returns to a new system.
11
22
Measuring costs and
returns
Financial metrics and
government accounting
What is measurable within a government
accounting frame is set by the generally
accepted and/or legally mandated
accounting standards and practices that
apply to the particular government
organization. Definitions and structures
for financial data are standardized and
designed to serve specific government
purposes. The Office of the New York State
Comptroller describes the state framework
in these terms:
“The purpose of classifying accounts is to
provide a standard format for recording
and reporting financial transactions
which allows comparisons to be made
with others municipalities or other
financial periods.”
“The classification system serves as a
basis for budgeting, accounting, and
reporting as well as for administrative
control purposes, accountability to the
Office of the State Comptroller and the
general public, cost accounting, and the
compilation of financial statistical data
on the state level. 11”
This framework organizes the accounting
data about costs/expenditures and revenues
according to general organizational
programs, functions or divisions, and by
objects of expenditure (e.g., salaries,
equipment, etc.). This is sometimes called
a function-object accounting or budgeting
system. A typical function-object accounting
system for a state agency has a highly
detailed and structured way of defining
what kind of financial information is to be
recorded and how it is to be organized.
Office of the New York State Comptroller. Accounting and Reporting Manual. Albany, NY (no date), p. 17.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
The strengths of a government accounting
framework for analysis of costs and
benefits are also its weaknesses, namely
standardization and rigid structures and
rules. Standards and rules make such a
frame-work useful for control of finances,
generating standardized reports, and
overall fiscal accountability. And if the
existing categories of accounts happen
to provide the particular cost and return
data needed for ROI analysis, the
frame-work would be readily available and
useful. However this is seldom the case,
particularly for IT investments. The full value
of an IT investment is typically a mixture of
many types of expenditures and other
costs, the accurate accounting of which is
not possible in the traditional accounting
system. These systems will usually record
the costs of equipment and personnel
attached directly and full-time to IT activities.
But they seldom record the changes in
personnel costs for those who use the
technology or the investments in training
and learning to operate new systems.
Maintenance and supplies may appear in
a range of separate accounts, and many
managerial and analytical costs are very
difficult to estimate or prorate accurately.
Measuring costs and
cost-effectiveness
In its most basic economic sense, a cost
is whatever you have to give up to get
something else. And a return is anything
good that you get as a result, whether that
good is measured in financial terms or
otherwise. This is, of course, a much
broader view of costs and returns than
is found in most accounting systems.
For example, an accounting system would
not likely include as part of the cost of a
new computer system the amount of time
devoted to the purchase decision. Nor
would the accounting system record the
benefits of increased staff morale from a
more functional or reliable system.
A full consideration of costs requires
attention both to opportunities and to
indirect costs. An opportunity cost
framework takes into account what
alternative actions or returns would be
missed or foregone as a result of a
particular investment decision. The path of
any IT project necessarily excludes other
paths not taken. If opportunities along those
foregone paths can be identified and
valued, they can become part of the cost
calculation. If I choose to implement a new
procurement processing system, for
example, the staff time spent training for
implementing the new system is time
unavailable for training in some other area
of work skills. If a value can be assigned,
this lost opportunity may be considered
part of the overall cost calculation. Indirect
costs (sometime referred to as imputed
costs) are those that are not incurred or
measured directly, but are calculated or
estimated from other measures. For
example, the cost of using existing network
cabling for a project could be calculated
from the amount of new traffic generated or
from some other prorating factor. Similarly,
apartment renters do not pay real property
taxes directly, but the renter’s portion of
that tax can be calculated as an indirect
component of rent.
These costs are often “off the books” in
the sense that typical public accounting
and financial management systems do
not provide this kind of cost analysis.
There are no standard accounting
practices to guide the measurement of
these costs. An agency may have an
existing standard indirect cost factor that
is used in budgeting for Federal grants or
contracts (e.g., some fixed percentage
of personnel expenditures). But these
standard factors are based on averages
over many projects and are not particularly
useful measures for any one particular
project. Opportunity costs are even
more problematic and are often not
included, leading to possibly significant
underestimation of overall costs.
Underestimating true costs can make
an investment’s returns look more attractive
than they may really be.
Assessing effectiveness requires
identifying the outputs of the project and
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
23
its implementation in business processes or
program terms. That means identifying
meaningful units of output that can be
related to the cost estimates. For example,
the effectiveness of a new system to process
business permit applications over the Web
could be measured in terms of increased
numbers of permits processed during a
given time. Lower costs per processed
permit would be a useful cost/effectiveness
indicator. Producing such a cost/
effectiveness measure requires detailed
data about permit processing and a way of
assigning costs on a per permit basis.
Without some enhancement, most government accounting systems do not provide the
basic data necessary for such a calculation.
It may be necessary to do detailed data
collection on a sample of permit transactions, for example, to establish a baseline
unit cost figure. A business process model
could provide the necessary data if so
designed. Even if such a unit cost measure
is available, it may have problems, such as
the distorted assumption that all permits
are equally important or costly.
Efficiency measures
Efficiency is a way of describing the
effectiveness of a project or system in
relationship to costs (or other inputs).
Efficiency cannot be separated from
effectiveness, since using resources and
failing to achieve a desired outcome can be
little more than waste; you don’t save money
by building half a bridge. Efficiency is
usually expressed in terms of optimizing the
value of a return for a given cost or input, or
alternatively minimizing the cost for a given
value of result. It is possible, of course, to
improve efficiency without necessarily
achieving an optimum. As long as it is
possible to compare cost/effectiveness or
return ratios for alternative systems or
methods it is possible to make judgments
about efficiency. To demonstrate an optimum
result or projection, it is necessary to have
a simulation or optimization calculation to
provide the data. Some operations research
and simulations, such as linear programming or workflow simulations, can generate
24
this type of calculation if the necessary
models and data are available. However
most IT projects do not have the data or
analytical resources to include these
methods in an ROI analysis.
Impact measures
The identification of variables that make up
a social cost or benefit calculation along
with their definitions is broader than both
the economic and accounting frames. They
are based either on the specific program
results desired by an agency or on general
social benefits and improved quality of life.
These impact measures can come from
several sources.
Operational data. Some useful measures of
social or broader economic outcomes may
be available in the data collected during
ordinary program operations. Providers of
services for homeless persons, for example,
routinely collect data about programs and
activities of clients that could be indicators
of impact on their quality of life or progress
toward independence. Similarly, law
enforcement agencies routinely collect
crime statistics that can be useful indicators
of neighborhood climate or quality of life.
Social and economic statistics.
Government agencies collect statistics on
social and economic indicators that are
relevant to the overall social and economic
status of their jurisdiction. These range from
the enormous resources of the US Census
Bureau to state and regional planning
agencies, and include market research
statistics available from private sources.
Special studies and evaluations. For large,
high cost projects separate data collection
activities, such as surveys or field research,
may be used to collect and analyze data
about social and economic impacts of a
project. These efforts may be expensive and
time-consuming, but may also be the only
way to obtain data about particular
outcomes. The case example in Appendix C
on Social Return on Investment provides
some examples of how this may be done.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
How time perspectives
change the measurements
Time has important effects on the
measurements and calculations. Both the
costs and returns of an IT investment
extend over time. Moreover, the costs may
be incurred long before the returns are
realized. So when estimating the money
value of costs and returns, it may be
necessary to take into account the effect
of the time perspective on the value of
money. That is, a dollar in hand today is
worth more than the promise of a dollar
in hand a year from now. How much less
that future dollar is worth, known as the
discount, depends on the current time
value of money, which is usually called
the interest or discount rate. Thus if the
discount rate is 5 percent per year, then
the promise of a dollar a year from now is
worth only 95 cents today. The farther out
in the future, or the larger the discount rate,
the less the present value of the promise
of that future dollar. Calculations based on
the discounted value of future returns are
useful to estimate what size of return is
needed to justify the costs of an investment.
The typical methods used in this kind of
financial analysis are called Net Present
Value and Internal Rate of Return
calculations. Both methods use an initial
cost and some assumed figures for a future
stream of returns, along with a reasonable
assumption for the interest (or discount)
rate. The Net Present Value (NPV)
calculation estimates the value today of
returns coming in the future, less the cost of
the investment. If the NPV result is less than
the cost, then the investment does not
appear to yield a net gain. The Internal Rate
of Return (IRR) calculation yields the same
sort of result by a different path. Using
the same basic information as the NPV
calculation the IRR calculation yields the
rate of return for the investment assuming a
break even, or NPV of zero. If the IRR is less
12
than the current interest rate, this indicates
that putting the investment in an interest
bearing bank account would yield a bigger
financial payoff than investing in the project.
An example of NPV and IRR calculations
are shown in Table 3 (page 26). This
example is based on a hypothetical
investment analyzed over a six year period.
It shows that when the time value of future
returns is taken into account, the total
returns must exceed the total cost by a
substantial margin to generate a positive
net present value or internal rate of return.12
The problem of choosing the
project or investment life cycle
Notice that any calculation of this sort
requires specifying a time-frame within
which the value of the project or
investment will be assessed. This time
frame is sometimes referred to as the life
cycle of the project. Any IT investment is
assumed to have a limited useful life,
beyond which the returns are not
considered, or when the expected returns
from some new technology will lead to
replacement of the previous investment.
For IT investments, the rapid change in
technology makes the choice of a
reasonable life cycle or planning framework
a difficult but necessary part of the analysis.
Risk analysis
in the public sector
Risk analysis consists of assessing the
importance of threats and how to mitigate or
eliminate them. We usually evaluate a threat
in terms of how likely it is to materialize and
how much damage or cost would result if it
did materialize. We know, for example, that
a large asteroid could hit the Earth and
destroy a continent—an enormous threat in
terms of consequences. But astronomers
The discount rate shown in these calculations was chosen for illustration only.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
25
Table 3. Net Present Value Calculation13
Year
Costs
1
$ 100,000
$
15,000
2
$
5,000
$
20,000
3
$
4,000
$
25,000
4
$
4,000
$
25,000
5
$
4,000
$
30,000
6
$
3,000
$
35,000
Total
Returns
$ 120,000
$ 150,000
Discount Rate
Net Present Value
Internal Rate of Return
tell us that the likelihood of that happening
is very, very remote—therefore, a small
threat in terms of probability. Because of
the low probability, the threat is seen as
small enough that few of us (aside
from some science fiction writers and
astronomers) take it into account in
day-to-day life. The same logic applies to
risks in IT investment. Consequently risk
analysis can be thought of as having four
basic steps:
I
identifying the source of threats,
I
assessing the extent of potential
damage or cost to the project,
I
assessing the likelihood of the threat
materializing, and
I
devising ways to reduce or eliminate
the threat (i.e., mitigate the risk).
3.0%
$
13,297
6.0%
Threats can be reduced by taking steps
to lessen the damage or cost that would
occur if the threat materializes and also
by reducing the probability that the
threatening event or action will occur. For
example, a project could call for using the
most reliable platform for a critical database
application, reducing the probability that
the system will experience a failure. The
project could also implement a backup
system capable of taking over processing
if the primary system does fail, reducing
the potential damage of a failure.
The overall subject of risk analysis is too
large to treat in detail here. However, most
of the risk assessment issues described
above involve problems of thinking beyond
the boundaries of the project, measuring
factors, or determining probabilities. This
should not discourage risk analysis.
13
Calculates the net present value of an investment by using a discount rate (rate) and a series of future payments and income (values) in each
period (i) for n periods.
26
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
The experience of those involved in IT
projects can be a rich source of intelligence
and experiential data on which to base
reasonable estimates of risk potential
and problem sources. The literature on
IT investment is another rich source of
analyses of successes and failures that
provide additional insights into risks and
mitigation strategies. Simply recognizing
where uncertainty and potential damage
lie is half the battle. Careful risk analysis,
based on the best available data and
estimates, will surely assist in ROI analysis
and improve planning, even if the amount
or quality of data is less than ideal.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
27
Appendix A: Case 1
Reducing the Cost of Web Site
Development and Maintenance
I
n order to better understand the problems
and issues of ROI analysis, the Center for
Technology in Government decided to use
a form of ROI analysis for one of its own IT
investment projects. The Center’s mission
to build and disseminate knowledge about
IT uses in government usually leads to
projects with other organizations. But this
case provided an excellent opportunity for
learning and knowledge building as part
of the Center’s own internal work. The
Center’s Web site is a critically important
tool in disseminating information and
maintaining contact with colleagues
and customers.
During the time covered by this case, the
Web site consisted of approximately 3500
pages and experienced over 1000 visitors
per day. The Web site was supported by
a full time Webmaster and part time
maintenance and development staff of two
professionals and two graduate students
who are part of the Center’s Technology
Unit. The Web site had been in operation for
several years and there was a substantial
baseline of experience with development
and maintenance using the current
architecture. Any investment in new Web
site architecture would use internal staff
and resources. Therefore, it presented a
good opportunity to apply ROI analysis to
a realistic problem in a situation where
data collection, analysis methods, and the
results can be examined at close range.
The purpose of the investment was to
change the Center’s Web site from a static,
HTML-based architecture, to a dynamic,
XML-based one. The problem to be
solved was a real one, namely that the
Center’s Web site had grown in size and
complexity to the point that maintenance
and additional development put a serious
28
strain on existing resources. In addition,
the static site could not support new
formats and capabilities the Center
wanted to use on the site.
The Center faced the decision of whether
to continue to invest additional human and
technical resources in the existing HTMLbased static architecture, or change the
architecture into an XML-based one that
had potentially much lower maintenance
and development costs. The change would
require a substantial initial investment in
tools and staff time to learn how to use
XML and related development applications.
The ROI analysis would help answer the
question of whether the potential returns
to be obtained by the use of a dynamic
(XML) Web site architecture would exceed
the costs of the conversion. The expected
returns would consist of savings compared
to the costs of maintaining and continuing
to develop the existing Web site. Returns
would also include the expanded abilities
of staff to create enhanced Web site
capabilities and features using the new
architecture and application tools.
Estimating the potential cost of expanding
the staff to maintain the current Web site
architecture was very straightforward.
However, estimating the conversion costs
to a dynamic Web site was a much more
complicated problem. A complete ROI
analysis should also include an estimate
of the savings (if any) to be obtained by
using the dynamic architecture. Estimating
the savings would require comparison with
the costs of maintaining and developing
the existing system, so estimating these
costs was part of the design.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Advantages of a dynamic
Web site
Web sites can be created with either a
static or fully dynamic architecture. Static
Web sites consist of HTML pages that
combine content (words, numbers, and
images), logic (how the content is
manipulated), and presentation (colors,
layout, fonts, and formatting). In HTML,
there is only limited capability to manage
or change the content separately from the
way it is manipulated or presented.
Managing and changing the material
requires working with content and HTML
tags directly within each of the pages. Static
sites are usually easier to develop than
dynamic Web sites, but can become much
more costly to maintain and manage as
they increase in size. Dynamic Web
architecture, in this case based on XML,
provides the ability to greatly simplify the
management, evolution, and expansion
of a Web site by providing control of the
logic and presentation independent of
the content. For example, users can be
presented with information based upon their
individual preferences, such as larger fonts
for a visually impaired person. The site can
also be made much more interactive.
Dynamic Web sites are generally more
expensive to develop than static ones,
but are cheaper to manage and maintain
up-to-date content.
Data sources
The best source of data about the costs
and returns for this new Web architecture
came directly from the staff’s experience.
The Center staff needed an estimate
of costs and returns for the XML
conversion that they could trust as a valid
representation of what they could expect in
a full-fledged conversion, without actually
implementing that conversion. To do this,
they chose to develop a few relatively
small new Web-based applications as
pilot projects using the new methods
and architecture. The first was a decision
making guide to help organizations design
electronic information access programs.
The project was called “Gateways.” The
experience from those efforts provided the
necessary information about costs and
returns to inform the larger decision.
Technology Unit staff members responsible
for Web site development and maintenance
consisted of three full-time personnel and
two half-time graduate assistants. All were
individually interviewed weekly for a period
of four months. During the interviews,
members were asked to recall the work they
had done the previous week. They were
asked to identify:
I
whether tasks were done in HTML or in
XML,
I
the amount of time spent in production
(“doing” the work) versus learning the
new application in order to do the work,
I
the benefits of working with XML, and
I
the drawbacks.
The data for the study also included
information about the operation of the Web
site and related activities, such as impacts
on routine site maintenance, changes in
content development procedures, and
coordinating work on this activity with
other technical tasks.
Weekly interviews were a low cost and
relatively unobtrusive data collection
method. They provided enough data so
that it was not necessary to observe work
directly or have the staff record their work
times in activity logs. This method fit one
purpose of the case: to be a useful example
for others contemplating ROI analyses,
employing methods that would be useful
in a wide range of settings without
requiring highly specialized training or
high costs. The staff involved in the work
agreed to the interviews and prepared
for them as a regular weekly activity.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
29
Cost estimation
One goal of the case study was to establish
an estimate of baseline costs for Web site
operations. This was complicated by the
fact that the CTG Web site is continuously
growing with the addition of new and
different types of information. The
estimated costs for maintenance of the
current site (at the start of the study period)
were approximately 75 percent of one
full-time staff person’s effort per week.
This was treated as stable over the term
of the case study.
The case data also included estimates of
the effort devoted by staff members to
researching XML (dynamic Web page
design), which began several months
prior to its initial use by the Technology
Unit staff. The director of the Technology
Unit had been contemplating moving
to XML for quite awhile. This early
deliberation was prompted by the rate of
growth for maintenance and development
efforts on the CTG Web site. The effort
devoted to this period of informal research
and deliberation was considered part of
the initial investment
Baseline costs. The Center’s Web site is
very extensive, consisting of approximately
3500 Web pages and receiving on average
over 1000 visitors per day. Information is
continuously updated and new products
and reports are introduced frequently.
Maintaining the Center’s Web site is quite
an undertaking. Under the old HTML
architecture, a Webmaster was responsible
for maintaining site links, updating current
materials, adding announcements and new
materials, and related tasks. A full-time staff
person devoted approximately 75 percent
of their effort to maintaining that site
(considered nine person-months per year.)
Investment costs.The primary investment
cost for this project was staff time to
learn the software and develop the skills
necessary to build the pilot projects. The
software costs for the conversion might have
been quite high, since initial research
30
showed most commercial products to be
very expensive. However, after a review of
available tools the Technology Unit decided
on the open source (free) application,
Cocoon. This tool seemed to fit the needs
of CTG and the staff began working with it.
While the software was free, it required
the staff to learn new skills. The underlying
logic of how the Web site operates was
also considerably different under the new
architecture. So the development team had
to learn to think about its tasks in a new
way as well. By contrast, there were zero
learning costs for maintaining the Web site
with the old architecture and methods.
The initial learning process was a slow
one. It took approximately three months for
the staff to reach a level of skill necessary
to move from using HTML to using XML.
At first, as much as 80 percent of their
time was spent on learning the XML
application and related skills, and 20
percent on production. After a 10-week
period, the time devoted to learning dropped
to approximately 40 percent of the total,
with the remaining 60 percent used in
production. Three months into the effort,
20 percent of staff time went to learning
and 80 percent to production. The staff
found that this latter 1:4 learning-toproduction ratio is consistent with the
learning-to-production ratio under HTML
(prior to the switch to the new technology).
The learning took place in two stages. In
the first stage, the team shifted its way of
thinking about Web development from the
HTML to the XML architecture. The second
stage was to learn the technical details of
working in XML. After the initial three-month
learning stage, staff members considered
themselves knowledgeable enough about
the language of XML to deal with problems
quickly. The second stage included learning
to use the Cocoon development tool. Since
documentation was limited, they used the
Cocoon listserv for questioning others
who use the program. Much of this
learning came from applying the new tools
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
to the tasks specific to the pilot projects.
This included creating interactive tools
that allow users to work with applications
on the Web site. The team’s learning
process was focused and task-specific,
the results of which could be applied
immediately. The staff avoided exploring
unneeded features and functionality of
Cocoon.
Additional learning costs are reflected
in the different amounts of time needed
to revise the printed version versus the
electronic version of the Gateways
decision making guide. The first pilot
application developed with the XML
architecture was based on that guide.
Initial changes to the electronic version
of the guide took up to 50 percent longer
than print-only revisions because of the
time spent learning to apply the new
technology in the Web application. Initially
the staff found it difficult to use XML data
structures. As a result it took considerable
time to understand basic concepts and
acquire basic skills for solutions that at
first often appeared to be quite simple.
Benefits
More efficient development of new
features and capabilities. Benefits were
seen almost immediately during the learning
process. During the first three months the
staff were able to see how XML would
directly benefit their future work. For
example, a separate file of interactive
tools was integrated into the Gateways
Guide using established style sheets
taken from existing components of CTG’s
Web site. The new technology saved
considerable amounts of time. This
particular task required only four hours of
work instead of the estimated 16 hours it
would have taken using HTML.
More efficient content management.
In HTML, the content of the page is tied
to the format, which makes it difficult (if
not impossible) to separate ongoing
development and maintenance functions
from content revisions. In order to update
the content of a page, some knowledge of
HTML is required. XML operates differently,
separating “content” from “presentation” or
“style.” Since the content and style are not
tied together, the ongoing development
and maintenance functions are separable—
allowing for division of labor or
specialization. Content “owners” or
“creators” work only with content while
Web designers work with Web coding.
In XML, content changes can be managed
by the content creator alone. Changes
made to the XML source file will appear
in all formats such as Netscape, Internet
Explorer, text-only, and PDF. Under HTML
changing one word of content typically
requires changing that word in all formats.
In comparison, XML saves considerable
time and ensures consistency throughout
the site. More importantly it allows for
greater flexibility in managing content.
Since changes are more easily made, it
is more likely that the information will be
updated and changed as needed.
When XML is used, the Web site can be
managed quite differently from HTMLbased methods. Work in XML and Cocoon
played to the strengths of the staff members
by allowing different components of the
Web site (content, logic, presentation)
and their related tasks to be kept separate.
Individual staff members can concentrate
on work at which they are more proficient.
Dividing the labor saves time and allows
staff members to become “experts” in
specific tasks. In an HTML environment,
all three components—content, logic, and
presentation—are all combined in the
individual Web pages. Anyone working
on the page must understand and deal with
the components together. Small changes
in content could require extensive work with
logic and presentation factors and involve
much more interaction and coordination
among content creators and HTML workers.
In the XML environment, content revisions
and expansion is separate from the other
components, which can be manipulated
largely independently. Content changes
require very little coding or programming
resources, and changing or developing new
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
31
programming capabilities is not constrained
by potential impacts on content.
Less maintenance effort. Current Web sites
based on HTML use the Web page as the
unit of composition; so a 50 page site can
be thought of as having 50 separate units of
work. These static Web pages combine both
content and formatting. As a result, changes
in one page have virtually no effect on the
others. For a single Web page or a Web site
with few pages or minimal updates, working
on individual pages may be relatively simple
and affordable. However, as the size and
complexity of a site increases, the cost of
ongoing development and maintenance
increases dramatically. It thus becomes
increasingly difficult to keep the site
consistent and up to date. In addition, some
functions, such as forms or databases, are
difficult or impossible to do on static sites.
In XML, code and content are easier to
maintain since code and content are stored
in separate sources. XML allows for the
streamlining of ongoing development and
maintenance functions because content is
stored in one place but propagated in many
places. Because of this architecture there
are fewer files to maintain (a 50-page site
may contain only one content file and 3–5
style files). Less time is required to maintain
the fewer files. One of the benefits of
employing this new approach, despite
the time and resources spent on the
learning process, comes from the potential
“payback” in greater functionality, easier
maintenance, and reduced ongoing costs.
Greater browser support. In XML, Web
design is independent of browser
capabilities. Changes made to an XML
source file will appear in all formats such as
Netscape, Internet Explorer, text-only, and
PDF. Under HTML changing one word of
content would require changing that word in
all formats. XML saves considerable
amounts of time. More importantly it allows
for greater flexibility in managing content.
32
This lowers the cost of adjusting formats
for presentation in different browsers. It is
difficult and time consuming to craft HTML
documents to work consistently in all
browsers, since each browser interprets
HTML standards differently. However, when
a page is created in XML using the Cocoon
framework, it can easily be made viewable
in all browsers. Staff no longer devote time
testing to ensure a page is viewable in all
browsers, since the inclusion of an XML
browser parameter automatically adjusts
formatting.
Greater platform support. XML standardizes and universalizes “content” or “data”
across platforms. Currently, a majority of
Web pages are mainly constructed using
HTML. The “content” is tied to the HTML
format. The page works on a Web browser,
but may not work on other platforms such as
wireless devices, cell phones, and PDAs.
XML offers greater delivery functionality
across a variety of platforms. It allows the
Center’s staff and customers the flexibility
of viewing pages of the Web site on any
Web server architecture.
More efficient management of style and
presentation. Style and presentation of the
Web site are more efficiently managed with
XML than with HTML for many of the
reasons discussed above. Since the page
itself is not the unit of composition, it is
easier to maintain consistency throughout
the site. Banners and footers, for example,
are maintained as single files and can be
imported to all style sheets. The separation
of “content” from “presentation” allows for
division of labor. Since Web design is
independent of browser capabilities, Web
pages are viewable in all formats.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Summary of costs and returns
The relationship between investment and
returns in the Web site conversion is best
expressed in terms of shifts in productivity
and opportunity. The principal costs of the
project are reflected in the opportunities
sacrificed and the added personnel costs
incurred in order to develop new skills and
capabilities. The primary cost component is
opportunity costs: staff were diverted from
routine work to learn new skills and to
begin to use the new techniques. Some
additional staff resources, in the form of
graduate students, were used as well.
The magnitude of that opportunity cost is
represented in the comparison between the
two charts in figures 4 and 5.
If the project had not been started (Figure 4)
there would be available resources during
the first three months of the time-frame to
maintain routine Web development and to
pursue some new opportunities. In the same
time period (Figure 5), by contrast, the
project began with a much expanded Web
development/learning effort that consumed
all slack resources and some operational
resources. No other opportunities were
taken up and ordinary Web operations were
reduced. That was a period of substantial
investment with little return.
By months 4–5 into the project (Figure 5),
sufficient learning had taken place and
allowed for exploiting new opportunities.
As experience and the stock of usable
components grew, the cost of added
development dropped rapidly and resources
were freed up to undertake even more new
opportunities.
Figure 4. Cost Trends
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
33
Figure 5. Cost Trends
The cost of ordinary Web operations did
grow somewhat during that period due to
the normal development of new content.
However, the total cost of including new
content and making routine maintenance
changes in the site is expected to drop,
despite increasing volume, due to the
increased efficiency of the dynamic
architecture. As efficiency in the
development and operations of Web site
work increase, the opportunities for
exploiting new opportunities grow. This can
be contrasted with the projections in
Figure 4 that suggest the increasing cost
of ordinary operations under the old
architecture will squeeze out development
opportunities and in time will likely exceed
the budget constraint.
34
This kind of opportunity-based analysis can
show positive results when relatively little
financial data is available, and when the
value of new products or capabilities are
difficult to determine. There are no savings
apparent from this effort unless the analysis
takes into account what it would cost to
develop these new capabilities and exploit
new opportunities under the old technology.
However, the expansion of development
opportunities over time, in contrast to the
baseline scenario, does give a reasonable
basis for estimating the returns for an
investment such as this in its early stages.
Since the projections are based on the early
stages of the work, it is possible that the
unanticipated problems of increased scale
could change the results over a longer term.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Appendix B
Case 2: ROI for Data Integration in
Health and Human Services
Project goals and the
context of State-Level
welfare reform
The information reported here was done for
a project designed to integrate information
resources used by Iowa agencies in the
administration of welfare programs and
welfare reform efforts. The project was a
response to the changed requirements for
the administration of welfare programs
resulting from the Personal Responsibility
and Work Opportunity Reconciliation Act
(PRWORA) passed by Congress in 1996.
The Act gave states welfare block grants
and increased discretion in the allocation of
these welfare resources. Iowa, as did most
other states, responded to this major policy
shift by creating its own new policies and
administrative arrangements. The project
analyzed here, the Welfare Reform Related
Technology Fund, was one of those
responses. The fund supported system
developments for the major welfare
programs in the State, such as Temporary
Assistance for Needy Families (TANF), food
stamps, and Medicaid.
Project rationale
In order to use the block grants efficiently
and to design their own human service
programs, Iowa emphasized accuracy of
eligibility determination, benefit distribution,
service delivery, and client support.
Accurate and timely information is a critical
resource for many complex decisions
required in administering this mixture of
benefits and requirements. Information
infrastructure and resources are necessary
to make accurate decisions and produce
higher quality programs.
In addition to the desire to enhance
program efficiency, the PRWORA included
financial sanctions for states that fail to
comply with regulations or to achieve
program goals. Improved information
resources and technologies were seen
as ways to enhance programs, achieve
efficiencies, and reduce the risk of
sanctions.
The sanctions can be considerable. If the
state did not meet the Federal requirements,
future funding for related programs could
be threatened. For example, failure to
comply with current Health Insurance
Portability and Accountability Act (HIPAA)
requirements can result in loss of Federal
Financial Participation (FFP) of 90 percent,
and penalties up to $25,000 per person,
annually, in addition to civil penalties. The
possible sanctions linked to TANF could
amount to as much as $14.8M per fiscal
year.
Investment in improved
welfare administration
Existing technology allowed the State
of Iowa to assist front-line workers in
determining eligibility and benefits, to
meet some Federal reporting requirements,
assist in program evaluation, and to make
information available for decisions
regarding program and related personnel
issues. To improve these information
resources, the state allocated a little over
$1 million in the 2001 fiscal year to the
Welfare Reform Related Technology Fund
with the available funding under TANF.
These funds were appropriated in the
Department of Human Services
Appropriation Bill. The IT program was
100 percent funded by the TANF block
grant funds.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
35
The funding supported a variety of IT
enhancements for welfare administration.
They included enhancing application
development and system programming,
tracking client eligibility, increasing data
storage and collection capability, and
implementing software updates and
changes. One of the expected results
of these investments was the capacity to
submit accurate and timely reports that
comply with the requirements for Federal
data reporting to the US Department of
Agriculture (food stamps) and Department
of Health and Human Services (Medicaid
and TANF). The state has to comply with
HIPAA requirements as well.
The information issues are large and
complex. The needs of each welfare client
may involve many agencies, each with its
own personnel, mission, and policies.
Each agency involved has its own IT
systems, producing many barriers to
information sharing and integration. None
of the staff in any single department was
able to access the complete information
about an individual client from existing
databases. Therefore, the investments
of this project were aimed at creating
standard protocols for data exchange,
a data warehouse, electronic referral
systems, resources directories, and
related applications.
Sustainable and coordinated hardware
and software development was required
to reach these project goals and provide
customers with better quality services.
The technology supported programs that
provide benefits and/or services annually
to approximately 20,000 families being
served by the Family Investment Program,
53,000 households receiving food stamps,
and 204,165 individuals receiving Medicaid
benefits (monthly averages).
14
36
Project methods
The beginning phase of the project was a
system evaluation. The goal was to provide
decision makers with a comprehensive
understanding of the beginning status quo
of the program. This would help avoid
wasting resources on system components
that were already functioning at a high-level.
The system evaluation was followed by a
search for existing software that could be
customized to meet project needs and
goals. The project planners then explored
the possibility of consolidating all the
information systems onto a single
platform to achieve integration and common
access. However, the implementation of
consolidation was judged to be too costly
and time-consuming, and so was rejected
as impractical. In order to retain the value
of existing legacy systems and infrastructure
investments, the planners chose to use
middleware as a more effective approach.
They required the middleware software to
be based on open standards and to extend
existing IT investments. The plan also
included replacing and upgrading
out-of-date hardware and ensuring that
the hardware is capable of supporting the
new applications and customized software.
Continuous staff training was included as
a key requirement.
ROI framework
All IT projects in Iowa agencies are required
to prepare ROI material in a standard
framework as part of any proposal for
new IT investment. The state provides a
Web site,14 supporting materials, and
applications for agencies to use when
preparing their proposals and ROI analyses.
The results reported here are taken from
those sources.
http://www2.info.state.ia.us/roi/index.html
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
The Iowa ROI framework requires attention
to certain standard components for all
proposals. These include the following.
Analysis requirements. Proposals
should include:
I
use of the Rapid Economic Justification
model 15 —to understand the business,
alternative solutions, cost-benefit
equations, possible risks, and
financial metrics;
I
cost and benefit measurements—
estimation of project costs and
benefits in some comparable unit
and determination of whether the
benefits exceed the cost; and
I
achievement of cost avoidance and
dollar matching from state sources.
Software. Proposals should deal with all
software components including:
I
application software;
I
operating system software;
I
interfaces to other internal and external
systems; and
I
standard protocols for data exchange,
data warehouses, linking software to
third party service providers, electronic
referral systems.
Hardware/Facility. Proposals should
deal with all physical components including:
I
additional platforms that accommodate
interoperable operating systems;
I
adequate storage and physical
environments;
I
adequate connectivity and bandwidth;
I
logical and physical connectivity; and
I
major interfaces to other systems, both
internal and external.
15
ROI results for the Welfare
Reform Related Technology
Fund
The expected benefits of the project were
to enhance organizational coordination
and reduce duplicate key-in and paperwork. Table 4 (page 38) is the financial
summary for the Welfare Reform Related
Technology fund for state fiscal years (SFY)
2001 through 2003. Total project cost (Row
A in Table 4, page 38) includes items such
as personnel, software, hardware, training,
facilities, professional services, supplies,
and others. Total annual project benefit for
the state (Row B) explains how much the
state has benefited from the investment of
the technology fund by the avoidance of
federal penalties. The magnitude of the
return on investment was expected to
decrease year-by-year. The likelihood and
magnitude of penalties would be largest in
the first year or two, and would diminish as
the accuracy and efficiency of the system
improved with experience and refinements.
This is a normal example of the operation
of diminishing marginal productivity of an
investment over time. At any rate, because
of the very large impact of trying to avoid
Federal penalties, the ROI percentage
remains very large. In addition to the
financial returns, the project planners
expect benefits resulting from more
efficient, effective implementation of
changes resulting in improved customer
service, increased program accuracy, and
readily available information for program
and field staff to use in making business
decisions.
http://www.microsoft.com/technet/treeview/default.asp?url=/technet/ittasks/plan/sysplan/wwww.asp
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
37
Table 4. Results of the IOWA ROI Analysis for Three Project Years 16
SFY 2001
$
B) Total Annual Project Benefit for State
$5,631,298,200 $529,600,000
$ 30,920,192
C) Total Annual Project Cost for State
$
$
E) Project Funds Requested from State:
(A*D)
760,300
$
1,468,324
SFY 2003
A) Total Project Cost
D) Project Funds Requested % from
State
2,024,768
SFY 2002
$
37.55%
$
F) Benefit/Cost Ratio: (B/C)
G) ROI: (B-C/ E)%
760,300
734,162
$
50%
$
734,162
1,556,016
778,008
50%
$
778,008
7406.7
721.4
39.7
740568%
72037%
3874%
** Some numbers differ from the original report due to rounding.
Potential risks in benefit
estimates
The benefit figures claimed in this analysis
appear to be based on a rather optimistic
scenario. The biggest return is cost
avoidance due to diminished Federal
penalties—a very large decrease between
the 2001 and 2002 fiscal years. These
penalty levels were very high for SFY 2001
and were expected to drop by over 90
percent in a single year due to the
introduction of improved information
technology. It is not clear from the available
documentation how realistic these penalty
reduction estimates were. A footnote to the
ROI analysis report for the SFY 2001 project
description reads:
“Avoidance benefits include $440,992
food stamp penalties, $5,600,000,000
potential Medicaid related losses, and
$29,592,824 TANF penalties. Funding for
the TANF penalties will be needed in
SFY 2002 ($14,796,412) and in SFY 2003
($14,796,412) plus Federal Match for
food stamps and Medicaid in the amount
of $1,264,384. There is additional
potential for sanctions due to food
stamp error rates. The amount of these
sanctions is unknown.”
The similar footnote in the FY2002
description contains essentially the same
estimates for all the other savings, but the
potential Medicaid related losses drop from
$5.6 billion to $500 million. This suggests
that estimates of this sort are subject to
considerable uncertainty and may not be
the best basis for an investment decision
without additional supporting data.
16
State of Iowa Return on Investment Program, IT Project Evaluation for Department of Human Services; SFY2001, 2002, 2003;
http://www2.info.state.ia.us/ROI/index.html
38
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Such optimistic estimates to justify a
project proposal are not unusual. In order to
win the resource competition, it is tempting
for agencies to assume the worst scenario
for not implementing the proposed project,
compared to the best case prediction for
completing the project. That way agencies
can show very dramatic and persuasive
returns for reviewers. Decision makers
have to find a reasonable balancing point
between the two extremes. Related
information needed to make more
reasonable assumptions may not be
available in the proposals, if it is not
required. Hence, the evaluation process is
usually problematic and critical. In this Iowa
Health and Human service case, reviewers
might need more detailed information about
the process of savings calculation in order
to make an accurate judgment. That is,
the proposers of a new investment may
deliberately skew their calculations to make
a stronger case than they could otherwise
justify. Reviewers may not be able to
detect such deliberate exaggerations or
unreasonable assumptions unless they
have full information about how calculations
were made. ROI calculations are products
of social, political, and economic interest
that are often in conflict with each other.17
The complexity behind the numbers and
calculation processes should always be
part of the overall decision making process.
17
Resources:
1. State of Iowa Return on Investment
Program, IT Project Evaluation for
Department of Human Services;
SFY2001, 2002, 2003; http://
www2.info.state.ia.us/ROI/index.html
2. Welfare Reform, Information Systems,
and the States, NASCIO; www.nascio.org/
publications/welfare1998
3. Government Technology, Case Studies:
Health and Human Services;
www.govtech.net/govcenter/solcenter
4. Microsoft’s Vision for Technology in
Health and Human Resources;
www.microsoft.com/business/industry/
gov
William Alonso & Paul Starr (editors), The Politics of Numbers. New York: Russell Sage Foundation, 1987
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
39
Appendix C: Case 3
Social ROI
The investment
philanthropy approach
This example shows the results of an
extensive effort to measure a wide range
of social and economic outcomes of
different investments in social programs,
in this case by a private foundation, The
Roberts Enterprise Development Fund.
The analysis of returns on the Fund’s
investment is based on impact-type
questions. The Fund invests by making
grants to nonprofit, community-based
organizations in the San Francisco area for
the purpose of “creating social value.” The
Fund refers to this approach as investment
philanthropy:
“Investment Philanthropy… is concerned
with the value accrued as the result of
charitable investments. Within this
perspective, social returns (that is,
benefits to society) generated by
philanthropic investments are the
measure of an investment’s success.
The critical challenge in Investment
Philanthropy is to compare the money
invested with the value it creates.18”
The Fund decided to evaluate the grants
to nonprofit organizations on the basis of
the social value created rather than on the
goals of the grantee or the apparent merits
of the organization’s efforts. To do so
required the development of methods to
define and measure that social value. Since
many government programs provide funds
for nonprofit organizations and the creation
of similar concepts of social value is the
goal of most government programs, the
methods in this case can be instructive.
The social return on investment
method
To implement the investment philanthropy
approach, the Fund developed methods
for measuring the social return on its
investments and created an administrative
mechanism to ensure that the analysis and
reporting information to assess SROI would
be available. The method defines social
value in a way that can be measured and
provides measurement procedures and
analysis techniques. The Fund requires its
grantee organizations to prepare and
submit SROI reports that provide the results
of these analyses to the Fund so that it can
evaluate the overall value of its investments.
The SROI method uses a definition of
value that covers a continuum from
purely economic to purely social, with
socioeconomic value in an intermediate
position:
Economic Socioeconomic Social
Economic value is represented by a
financial return on the Fund’s investment,
reported as increased revenue, asset value,
etc., from the grantee’s accounting report.
This financial return is defined and
measured according to the techniques
for accounting and demonstrating profit
creation that apply in the regular capital
markets—i.e., the stock markets and
private sector accounting methods. The
SROI reports prepared by grantees include
detailed financial statements equivalent
to those produced by publicly traded
companies in the for-profit sector.
18
Roberts Enterprise Development Fund. SROI Methodology: Analyzing the Value of Social Purpose Enterprise Within a Social Return on
Investment Framework. San Francisco: The Fund, 2001, p. 10.
40
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Socioeconomic value is defined by creating
methods to assign money values to social
outcomes wherever possible. For example,
increased employment opportunity for those
who work in grantee organizations is a
social value. Socioeconomic value is
expressed in part by the increased taxes
paid by those employed, as well as by
reductions in welfare costs. Another social
outcome of employment in grantee
organizations, reduced criminal activity,
is estimated by comparing arrest and
conviction rates for employees with similar
segments of the population. Reductions in
arrests and convictions are monetized by
counting the savings to the society of fewer
convictions and incarcerations. An example
of the calculations for social cost savings is
shown in Table 5.
Table 5. Calculation of Average Social Cost Savings Per19
Total
Total
# of Target
Average
Decrease
Decrease
Employees
Cost
(Increase) x Cost Per = (Increase) ÷ Responding =
Savings
in Annual
Visit/Use
in Annual
to
Per Target
Visits
Cost
Question
Employee
Public Assistance Programs*
TANF
NA
NA
= $
- ÷
20
= $
0.00
General Assistance
NA
NA
= $
300 ÷
20
= $
15.00
Food Stamps
NA
NA
= $
3,362 ÷
20
= $
168.10
SSI
NA
NA
= $
5,004 ÷
20
= $
250.20
Social Service Programs
Food Banks
1,050
x
$
26 = $ 27,300 ÷
20
= $ 1,365.00
Case Management
980
x
$
41 = $ 40,180 ÷
20
= $ 2,363.53
Community Clinics
34
x
$
86 = $
2,924 ÷
20
= $
222
x
$
176 = $ 39,072 ÷
20
= $ 1,953.60
1,650
x
$
62 = $102,300 ÷
20
= $ 5,115.00
Emergency Room
2
x
$
211 = $
422 ÷
20
= $
23.44
Legal Services
4
x
$1,029 = $
4,116 ÷
20
= $
228.67
Sustance Abuse Treatment
2
x
$8,060 = $ 16,120 ÷
20
= $
806.00
10
x
$3,761 = $ 37,620 ÷
20
= $ 1,881.00
Mental Health Treatment
Housing Services (shelter, trans,
housing, grp home)
MediCal (includes employee and
dependents)
146.20
Criminal Conviction Savings
$ 1,327.43
Average Social Cost Savings
Per Employee
$15,643.67
19
Roberts Enterprise Development Fund. SROI Methodology: Analyzing the Value of Social Purpose Enterprise Within a Social Return on
Investment Framework. San Francisco: The Fund, 2001, p.32.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
41
Social value is defined as the outcomes
for society as a whole that are positive,
but cannot legitimately be given monetary
value. For these measures, alternative
methods are used to generate some useful
measure of return in nonmonetary terms.
For example, a quality of life survey is part
of the data collection method for employees
and clients of the grantee organizations
(see Figure 6 below).
While such a survey does not yield
monetized values for an investment return,
it does provide meaningful evidence of
some social benefit. It has all the usual
limitations of survey research data, but
can supply evidence of social value
creation that would not be available by
other means.
Figure 6 - Survey Sample for Social Return Measurement 20
How I Feel About My Life
Strongly
Agree
Agree a
Little
Neither Agree
nor Disagree
Disagree a
Little
Strongly
Disagree
No
Answer
There are a lot of people I like
to hang out with.
K
K
K
K
K
K
I like to get together with friends
as much as possible.
K
K
K
K
K
K
I have people in my life who
really care about what’s
happening to me.
K
K
K
K
K
K
If for some reason I were put in
jail, there are people I could call
who would bail me out.
K
K
K
K
K
K
K
K
K
K
K
K
K
K
K
K
K
K
K
K
K
K
K
K
If I were sick or hurt and and I
needed someone to take me to
the hospital, I would have no
trouble finding someone.
If I were hungry and had no
money to buy food, there are
people I know who would give
me food.
If I were in trouble and some
people were going to try to hurt
me, there are other people I
could get protection from.
20
Roberts Enterprise Development Fund. SROI Methodology: Analyzing the Value of Social Purpose Enterprise Within a Social Return on
Investment Framework. San Francisco: The Fund, 2001, p.72.
42
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Measuring value and returns
The SROI methodology employs six stages.
The first three deal with measuring value
and the last three deal with the index of
return.
I
Stage 1: Calculate enterprise value
(uses standard accounting value
measures).
I
Stage 2: Calculate social purpose
value (assigns monetary values to
social outcomes).
I
Stage 3: Calculate blended value
I
Stage 4: Calculate enterprise index of
return.
I
Stage 5: Calculate social purpose index
of return.
I
Stage 6: Calculate blended index of
return.
An example of the overall results of these
calculations is shown in Table 6 on page 44.
These return calculations and the
supporting material are key components of
the SROI reports produced by the grantee
organizations on an annual basis. The
reports include SROI metrics, business
data, social impact data, and provide
analysis in these areas. Repor t contents
include:
I
descriptions of the social purpose
enterprise,
I
financial analysis of the social purpose
enterprise,
I
profiles of the enterprise’s target
employee population,
I
SROI metrics and analysis,
I
description of the nonprofit agency and
its mission, and
I
key social impact findings and analysis.
Overall, the SROI Report can be viewed
as a nonprofit organization stock report. It
provides a standardized way of estimating
value and presenting return calculations in
a clear and accessible manner.
Limitations in the SROI
approach
Some of the comparative indicators
available for evaluating for-profit
organizations are not available for this
SROI approach. There are no comparable
industry ratios and analyses for the
nonprofit sector, although research in this
sector is growing. In addition, the logic of
management in for-profit organizations is
different in crucial ways. Profit-maximization
strategies are not necessarily useful
or productive in the nonprofit sector.
Consequently, careful attention should be
given to assessing the appropriateness of
the standard financial measures to this
alternative use.
It is useful to consider the Fund’s
description of lessons learned over the
course of applying the SROI methods.21
The SROI analysis process is resource
intensive. The average practitioner must be
aware of the financial and human resources
necessary when conducting an SROI
analysis of their social purpose enterprise.
Considerable resources were provided
to support the work in the grantee
organizations.
Engaging the practitioner is essential.
This is not a “top down” process. It is
imperative that practitioners themselves
drive the process of identifying and
setting the social indices by which
they will assess the value of their life’s
work and the returns generated by the
investments they receive.
21
Roberts Enterprise Development Fund. SROI Methodology: Analyzing the Value of Social Purpose Enterprise Within a Social Return on
Investment Framework. San Francisco: The Fund, 2001, p.62
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
43
Table 6. Social Return on Investment Results 22
SROI Results
1999
SROI Metrics
Enterprise Value
Index of Return
$411,906
0.93
Social Purpose Value
$20,861,055
47.14
Blended Value
$21,222,960
47.96
Investment to Date
$442,543
Social Purpose Results (Per Target Employee)
1999
Public Savings
$15,644
New Taxes
$1,815
Wage Improvement
$12,097
Financial Improvement
$9,849
Enterprise Financials
Sales
1998
1999
2000P
$233,004
$537,789
$708,957
70%
69%
65%
5%
-4%
-4%
82%
10%
0%
Gross Margin
Operating Margin (Before S&S)
Operating Margin (After S&S)
Projected Values
Total Projected Investment
Total Projected Social Savings and New Taxes
Total Projected Social Expenses
Total Projected Contribution to Parent
22
44
(1999 into Perpetuity)
$575,775
$22,434,361
$1,573,306
$0
Sample SROI results accessed from Roberts Enterprise Development Fund’s Website at www.redf.org/pub_sroi.htm#methodology.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
Appendix D
Additional Resources
Enterprise architecture
Enterprise Architecture Development Tool-Kit, Version 2, National Association of State Chief
Information Officers, July 2002. https://www.nascio.org/hotIssues/EA/.
The Business Reference Model, Version 1.1, Federal Enterprise Architecture Program
Management Office, 2002. http://www.feapmo.gov/feaBrm.htm.
State-specific Enterprise Architecture initiatives
Connecticut: http://www.doit.state.ct.us/policy/domain/index.htm
Kentucky: http://www.state.ky.us/kirm/arcstand.htm
North Dakota: http://www.state.nd.us/ea/
Stakeholder analysis
Making Smart IT Choices. Center for Technology in Government. University at Albany-SUNY.
http://www.ctg.albany.edu/resources/abstract/absmartit.html.
John M. Bryson. Strategic Planning for Public and Nonprofit Organizations, Rev. Ed. San
Francisco: Jossey-Bass, 1995.
Business process analysis and modeling
Business Process Resource Center, University of Warwick, UK
http://bprc.warwick.ac.uk/umist1.html, http://bprc.warwick.ac.uk/index.html#BPRCTOP.
The Workflow Handbook 2002. Published in association with the Workflow Management
Coalition (WfMC) Edited by Layna Fischer. March 2002, 428 pages. ISBN 0-9703509-2-9.
Workflow And Reengineering International Association http://www.waria.com.
Workflow/BRP application vendors: http://www.waria.com/databases/wfvendors-A-L.htm.
Business Process Management Journal: http://www.emeraldinsight.com/bpmj.htm.
Formal modeling
Agent-based modeling
Center for Computational Analysis of Social and Organizational Systems
http://www.casos.ece.cmu.edu/home_frame.html.
Gaylord, R.J. and D’Andria, L. (1998), Simulating Society: A MATHEMATICA Toolkit for
Modelling Socioeconomic Behavior. Springer: New York, NY.
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS
45
Ilgen, D. R., & Hulin, C. L. (Eds.) (2000). Computational modeling of behavior in organizations:
The third scientific discipline. Washington, DC: American Psychological Association.
Michael J. Prietula, Kathleen M. Carley & Les Gasser (Eds.), (1998) Simulating Organizations:
Computational Models of Institutions and Groups, Menlo Park, CA: AAAI Press/The MIT Press.
Thomas Schelling, “Dynamic Models of Segregation,” Journal of Mathematical Sociology 1
(1971), 143-186.
Thomas Schelling, Micromotives and Macrobehavior. New York: WW Norton & Co. Inc., 1978.
UML modeling
Object Management Group. OMG Unified Modeling Language Specification, Version 1.3.
Framingham, MA: OMG Headquarters, 1999.
Doug Rosenberg, Kendall Scott (Contributor). Use Case Driven Object Modeling With UML:
A Practical Approach (Addison Wesley Object Technology Series). New York:
Addison-Wesley, 1999.
Workflow modeling
Thomas W. Malone, et al. Tools for inventing organizations: Toward a handbook of
organizational processes. Management Science 45(3) pp 425-443, March, 1999.
MIT Sloan School Center for Coordination Science. http://ccs.mit.edu.
Workflow Management Coalition. http://www.wfmc.org.
INSEAD Workflow Site.
http://www.insead.fr/CALT/Encyclopedia/ComputerSciences/Groupware/Workflow/.
Operations research
Michael Trick’s Operations Research Page. http://mat.gsia.cmu.edu/
INFORMS: Home Page: Institute for Operations Research and the Management Sciences.
www.informs.org/.
Operations Research Society. www.mors.org/.
MIT, Operations Research Center. http://Web.mit.edu/orc/www/.
46
CENTER FOR TECHNOLOGY IN GOVERNMENT—RETURN ON INVESTMENT IN INFORMATION TECHNOLOGY: A GUIDE FOR MANAGERS