Technology
Iamb ...
Sinan Isitan
SLA metrics are at the heart of successful
outsourcing agreements
Service level agreements (SLA) and their measurement play a
­major role in the realization of effective outsourcing management.
If the strategic goals of the SLA and the metrics for measuring the
goals are aligned, this carefully considered selection supports goal
achievement.
64
Detecon Management Report • 4 / 2011
Iamb or Dactyl?
... or Dactyl?
elecommunications network operators base their decisions
T
for outsourcing business processes on various strategic goals.
Generally they are related to cost-cutting efforts or the creation
of more breathing space so that the company can c­ oncentrate
more closely on the marketing of services to end customers.
­Examples range from the outsourcing of the maintenance of the
company’s software applications to the complete outsourcing of
network operation. Once such a decision has been made, the
commitment to this course continues for a number of years as a
rule and is manifested by a contract between the client and the
outsourcing provider. So-called service level agreements must
be concluded to ensure that this relationship functions well
and that the services which are supposed to be performed are
­actually carried out.
Service level agreements (SLA) are an essential component
of any outsourcing deal. They define the outlines of the project in terms of the tasks which are to be carried out and the
scope of the delivery which the service provider must perform
for the client as well as the quantity and extent of equipment
which will be purchased and delivered, the acceptance criteria
for the ­responsiveness, and the quality of the delivered product. ­Properly drafted SLAs describe in highly precise terms the
e­ xpectations of both parties as well as carefully defined metrics
for measuring the achievement of these objectives.
The heart of any efficient SLA is the performance metrics. ­These
metrics will be needed for the duration of the outsourcing
­relationship to measure the service provider’s performance and
to determine whether he is fulfilling his contractual obligations.
The selected metrics should meet certain criteria:
• They should measure the right performance characteristics to
ensure that the client receives the services at the promised level
and at the same time make it possible for the service provider to
realize an acceptable profit.
• Their implementation should be as simple as possible, ­detailed
to a reasonable degree, but without incurring unreasonable cost
expenditures.
• They should merge all of the contractual obligations into
­reasonable and achievable performance levels so that “good
­service” can be clearly distinguished from “poor service” and
give the service provider a fair chance to satisfy his client’s
­expectations.
65
Detecon Management Report • 4 / 2011
Technology
As we can see, the selection and implementation of suitable
­metrics for SLAs is of critical and decisive importance. The fundamental question: How can I adapt the client’s business goals
to suitable SLA metrics or reconcile these goals with the metrics?
The selection of suitable metrics to fulfill these criteria is a
­decisive preparatory step for any outsourcing contract.
Frequently a broad range of various metrics will be required
for the management of the many different aspects related to an
­outsourcing project.
While some metrics can be highly specific to a certain project,
there are also very many which are very typical and applicable
in general situations, making them appropriate for outsourcing
projects of many different types.
Certain metrics can function extremely well in one project while
proving to be completely ineffective for another project, simply
because the SLAs in the second case are too imprecise or require
too much effort for measurement. The unsatisfactory selection
of metrics quickly leads to SLAs which are almost impossible
to implement or do not even bring about the desired behavior.
In extreme cases, such circumstances can even end up as the
subject of legal disputes.
So this selection can be a complex process requiring a lot of
time and effort. Owing to the tremendous number of p
­ ossible
metrics, the organizational and historical experience, the
­
­behavior which is to be achieved, and the expenditures or costs
related to the collection of the measurement values should be
taken into account when making the selection. Use of common
sense is welcome to play a dominant role here! One should never
lose sight of the actual goal – to secure successful and p
­ ositive
collaboration between the parties.
Basic principles for the selection of suitable metrics
Five principles have proven to be useful when selecting metrics
appropriate for achieving these goals.
Clarity about the behavior which is intended
The most important goal of an SLA should be to induce a
desired behavior on both the client and the provider side. Normally, both parties will seek to optimize their individual performance goals with their actions. The selection of the “wrong”
metrics can very quickly cause good contractual relationships
to go sour. For instance, the measurement of maintenance and
the repair of disruptions of radio base stations based solely on
66
Detecon Management Report • 4 / 2011
the volume of the work carried out on the network elements
will create a misleading incentive. This type of work might
­unnecessarily be c­ arried out often and in a large volume w
­ ithout
actually i­ mproving actual network availability. The cause l­ eading
to ­frequent downtimes is possibly not remedied by this type of
work or there is no incentive to remedy the actual cause because
this would reduce the volume of work to be carried out and
subsequently the billing volume. An important fundamental
condition for motivation of the correct behavior is that each side
understand the other side with regard to expectations and goals
and that everyone understand clearly what factors are actually
under his control. The client, for example, should understand
that the service provider wants to make a profit from the business, while the service provider can expect the client to aim for
cost control, for instance.
The metrics set forth in the SLA should concentrate on the
desired target behavior. What incentives should be provided?
Cost reduction? Reduction of the error frequency? Fastest possible adaptation of network capacity? Where can concessions
be made, where should improvements be made? When suitable
metrics are being selected, the first step should be an examination of whether they are aimed at an improvement in performance of the desired behavior.
A tried and proven method is to step into the shoes of the other
party and to test the proposed metrics from his perspective.
What would you yourself try to do to optimize the performance
of the proposed SLA? Be creative. You should ask yourself the
question whether an optimization of the values will also lead to
the desired results. You will often need the so-called secondary metrics as “checks and balances” so that you avoid heading
down the wrong path. You should also make sure that the metrics can be defined as objectively as possible so that the room
for varying interpretations is as limited as possible. Whenever
metrics can be the object of subjective observations, differences
can quickly arise when it is a matter of deciding whether the
service provider has fulfilled his obligations.
Assurance that the metrics are actually under the control
of the service provider
The assurance that the service provider can actually control his
compliance with the SLA is of fundamental significance. The
service provider should make sure that the SLAs are bilateral.
If the service provider’s ability to achieve his goals is dependent
on the client’s behavior or on that of another contractual party, a means of measuring the external influence as well should
be provided. Conversely, no SLA metrics should be used which
dictate to the service provider how he must perform his work.
Iamb or Dactyl?
After all, we can assume with a clear conscience that the s­ ervice
to be provided in this case belongs to the service provider’s
core competencies and will embody many years of “best practice” ­experience. Any and every attempt to manage the processes from the outside will do nothing but create unnecessary
­inefficiencies. Instead, you should concentrate above all on the
expectations concerning time, cost, and quality for the services
which will be provided.
The measurements of the SLA should be simple to collect, using an
automated process whenever possible
If it is not possible to collect the data for the SLAs without a lot
of time and expense, the acceptance level will fall ­quickly and
may even be ignored completely at some time in the f­uture.
In the long run, no one will make the effort to collect data
­manually at high cost. Ideally, it should be possible to automate
all of the measurements and to use processes running in the
background with minimal overhead. You should not forget that
this will require certain investments in tools, so you need to
strike the necessary balance when making your selection. Often
alternative metrics which can be measured with lower expenditures will achieve the same results.
Less is more
The number of metrics should be kept as small as possible.
When people are drawing up new SLAs, it is not unusual for
them to give in to the temptation to define a very high number
of metrics which will later result in the generation of excessive
data. The belief that the more measurements or control points
included in a process, the better the control over the service
provider almost never proves feasible in practice. The better approach is to select a group of measurements which can ­easily
be analyzed, processed, and applied so that the project can be
managed effectively. If an unwieldy quantity of reports and data
is generated, the temptation will be great either to ignore the
reports completely or to be selective when choosing data so
that they can be interpreted subjectively for specific purposes, a
­situation which runs counter to the original purpose.
Selection of reasonable target values
The selection and definition of the metrics is only half the battle.
If they are to be of any use, reasonable and achievable target
values must also be defined. It is frequently difficult, especially
at the beginning, to determine suitable values – especially if the
client has not yet calculated the performance values himself and
there are consequently no historical data available. Companies
and organizations which previously had an active program for
the measurement of their organization’s performance have an
easier time of it. If this is not the case, we recommend conducting an examination of the relevant elements before beginning
the project so that you have a solid basis to use as a reference.
Ideally, a corresponding process should also be defined in the
Figure: Outsourcing Engagement Model
Response time
Volume of work
INPUT
(Defects, Requests,
Orders)
Task 1
Task 2
Internal reproduce
Informal
Quality
Task N
Acceptance criteria
Formal
OUTPUT
(Volume
fully oriented
work)
Internal costs
Costs/Efficiency
Source: Detecon
67
Detecon Management Report • 4 / 2011
Technology
SLA describing how the target values are to be flexibly adjusted
in the future so that performance can be improved successively.
Modeling of an outsourcing measurement
The figure below shows a model which helps to understand the
various factors which affect an outsourcing contract. Questions
such as “Why are certain categories of metrics needed, and why
should they be given preference in consideration if the performance targets for these metrics are defined at the beginning?”
are at the forefront.
The figure illustrates the types of measurements required to
support an outsourcing contract in its simplest form. It views a
contract as a black box which accepts a given volume of queries
as its input and produces a volume of services as its output.
The time required for the processing is frequently referred to
as the responsiveness. The work leads to costs so efficiency can
be defined in this case as costs per unit produced. Quality is
understood as the capability of a produced service which fulfills
certain acceptance criteria. Each of these factors represents an
interface between the service provider and the client and can be
affected as part of an SLA.
Certain factors are completely under the control of the c­ lient.
For instance, the client largely determines the volume of
­queries, the input. These queries can be submitted formally
and follow the standardized processes or be of informal nature
so that they ­never enter the official system. Direct instructions
from the client’s personnel to the service provider’s personnel
are ­examples of the latter. The recording of such queries, which
more or less fall through the cracks, is difficult, but ­nevertheless
necessary, and a genuine challenge for the definition of the
SLA. Since these types of informal queries are not officially
­sanctioned, they remain invisible to the pertinent client manager of the service provider and are also not covered by the SLA.
This is often a ­reason for the client’s later dissatisfaction with his
service provider.
Another important factor is related to previously existing ­defects
which were already present before the pertinent outsourcing
­agreement entered into force. These previously existing defects
68
Detecon Management Report • 4 / 2011
(in the client’s processes, for example) in the applications which
must be serviced or in the network infrastructure which is to be
operated cannot really ever be avoided completely. The ­legacy
defects, however, have significant impact on the ability of the
service provider to keep the contractual promises of quality.
The remedy of these defects can certainly be negotiated in the
outsourcing agreement of the particular case. But a smart and
experienced service provider will want to make a catalog of these
defects before committing to any contractual obligations.
Responsiveness, efficiency, and produced service volume, on the
other hand, are under the service provider’s control as the party
performing the service while the client, in his function as the
determinant, will usually define the standards for the acceptance
of the service. It is the service provider’s responsibility to calculate whether he is able to meet these requirements and still
make a profit. However, the service provider can usually control
only his efficiency, not the volume, unless the volume of services
to be performed is previously defined. Backlogs are the consequence if the volume of work orders is higher than the estimated
capacity, i.e., the maximum volume of incoming orders which
can be processed. Subsequent improvement becomes necessary
if the provided services from the service provider do not conform with the quality requirements and must be reworked.
Factors within the outsourcing black box include the number
of work steps, the efficiency with which these steps are carried
out, the volume of internal subsequent improvement work,
and the labor costs. These factors are located within the service
provider’s sphere of influence. Changes in these parameters by
the service provider affect his costs, his capacity, the responsiveness, and ultimately his profitability. So these parameters are
extremely significant for the service provider’s success, but are
generally not components of an SLA.
Categories for SLA metrics
There are various metrics which can be selected to manage the
above-mentioned factors. One simple approach is to sort these
metrics according to categories and then to decide what category is best suited for a given project. In the second step, the
specific metrics for the SLA are drawn up.
Iamb or Dactyl?
The factors which must be measured can as a rule be modeled
under four categories for SLA metrics.
Work volume
The work volume is an important measurement for the
­dimensioning of the outsourcing project. Generally ­speaking,
any work going beyond the agreed volume will be billed
­additionally by the service provider or will prompt a renegotiation of the SLAs. The general definition is usually expressed as a
number of units of the work product or service.
Quality
Quality metrics can vary greatly and are highly dependent on
the nature of the delivery product. If the work being delivered
does not fulfill the acceptance criteria as described in the applicable specifications and standards, quality problems will appear.
In the ideal case, every one of the delivery products described in
the SLAs should contain at least one quality criterion describing
the quality of the product to be delivered. The quality criterion
can be a composition of a number of individual metrics which
in their sum describe the acceptance criterion for the delivered
product, but it can also be defined by a single measurement.
­Examples of such quality metrics are downtime rates or error
rates, compliance with technical specifications or standards,
technical quality, service availability, customer satisfaction, or
the subsequent improvement rate.
Responsiveness
The responsiveness measures the time required by the service
provider to process a work query. The client frequently views
this as extremely important, i.e., the pertinent value has an
­extremely great impact on the general customer satisfaction.
Indeed, it is not at all unusual to find that an improvement in
responsiveness is a major motivation for the decision of some
network operators to outsource some of their work at all.
Efficiency
Efficiency metrics measure the capability of the outsourcing
contract to secure the required service at reasonable cost. In network operation, for example, this can be the costs per network
element.
Reporting – as simple as possible
Once the delivery product has been specified and the a­ ssociated
quality has been described in the form of metrics in the SLAs,
client and service provider must conclude additional agreements regarding the reporting of the relevant information
and measurement results for the duration of the outsourcing
contract. As is so often the case, the rule here: the simpler, the
­better. The key to effective reporting is to present the results in
succinct form. Instead of displaying long series of tables and
figures, the results should be summarized as trends, for instance.
Methods such as the balanced scorecard, the weighting of individual values within the context of the supraordinate project
goals, are often practical. Otherwise there is a risk that a single
poor value can trigger overreactions even though the general
trend may be positive.
Usually, the parties specify templates for the various reports
as components of the agreement as well as the frequency with
which the various report types should be prepared. In many
cases, this process of format specification causes the parties to
rethink the metrics which have already been agreed and possibly
to eliminate some of the measurements as superfluous. A rule
of thumb can be stated like this: if a measurement is not regarded as important enough to make a contribution or represent
significant benefit in a report, then it is usually not worth the
while to take the measurement at all. Depending on the scope
of a project, it is thoroughly possible that one single report containing all of the defined metrics will be completely adequate.
Sinan Isitan is Managing Consultant in the Group Programme Management
and Engineering. He has more than 15 years experience in the telecommuni­
cations industry as technical project manager and worked in several telecommunications outsourcing and infrastructure projects in Europe, Africa and
­Southeast Asia, both on the customer and on the supplier side.
[email protected]
69
Detecon Management Report • 4 / 2011