Recognize Progress: Measuring
Outcomes Not Outputs
Yvonne Attard, Director, Customer
Development, Oakville Public Library
Rebecca Jones, Partner, Dysart Jones
Associates
Agenda
•
•
•
•
•
•
Basis for discussion
Measurement System: On Overview
Critical success factors
Options
Can you really measure impact?
Measurement Framework – Theory and Reality:
Oakville example
• Gaps and hurdles
• “Libraries change lives”
Basis for discussion
• There is no one magic measure
• There IS a strong correlation between an
organization’s success and:
• Its clarity of purpose
• Its ability to understand its culture and its stakeholders’
• A well-designed performance measurement system that
fits that culture.
• “Value” is client or stakeholder defined
• Definition is dynamic
• It is two-dimensional: economic & psychological
• It is relative to alternatives
Basis for discussion
• Measuring for Results: The Dimensions of Public
Library Effectiveness by Joe Matthews, 2004
• Asserts that few public libraries have a “culture of
assessment”
–
–
–
–
Difficult and complex
Most measures indicate past performance
No cause-and-effect relationship between measures
Performance measures measure quantitatively, but library
outcomes are largely qualitative
Measurement System: an overview
Input
Output
Impact
Outcomes
Measurement System - Input
• Inputs are basically resources or capabilities:
• Financial resources*******
–
–
–
–
Content
Staff
Technology
Facilities
• Ultimately, who enables you to obtain these
inputs?
Measurement System - Output
The service or program created by using the
Input.
Measurement System - Outcomes
What the client is able to do with the service or
product; how the client uses the Output.
Measurement System - Impact
The effect or influence of the service or
program.
What changed for the client and/or the client
organization.
How the library changed a life or a group of
lives.
Measurement System - Example
Input
$100K collection budget
Output
2500 new resources in collection
Outcomes
75% of students in the community
schools used collection resources
Impact
% of students graduating rose 5%
in the past 2 years; EQAO results
improved in 3 of the 5 community
schools
Measurement System - Oakville
Input
$20,000 spent on an outreach program
to students budget
Output
2,500 students reached
Outcomes
1,500 new student library cards issued
during outreach program and increased
use in online resources
Impact
Students assignment grades increase,
ease of research, teachers view marked
improvement in studies
Systems View
Stakeholders
I
n
p
u
t
s
I
m
p
a
c
t
Feedback
Library
Clients
Outputs
Critical Success Factors
For defining, measure & communicating progress &
impact?
• Alignment of stakeholder goals and your goals
– Balance of government demands vs. business plan vs. library
demands
• Your definitions, measures & communication style match
those of your stakeholders, other service organizations,
non-profits and others in the local community
• “Doable” capture and analysis of measures
• Measures communicated in a meaningful and
understandable way for stakeholders
• Conversations outside the library
Some Options
• Balanced Scorecard
• LibQual
• Cost-benefit
Balanced Scorecard
• Aligns measures with strategies to track
progress, reinforce accountability and
prioritize improvement opportunities
• A “system” of measures based on 4
perspectives:
–
–
–
–
customer
internal
financial
Innovation
• Limits measures to those most critical
Balanced
Scorecard
Customer Perspective
How do we look
to our clients?
Goals
Measures
Innovation Perspective
How can we improve
& create value?
Goals
Measures
Internal Perspective
What must we
excel at?
Goals
Measures
Financial Perspective
How do we look to
our funders
or stakeholders?
Goals
Measures
Kaplan & Norton
Balanced Scorecard: Example
METRIC U.1.a. OVERALL RATING IN STUDENT AND
FACULTY SURVEYS.
Target1: A score of at least 4.00 (out of 5.00) from each of the
major constituencies: undergraduate students, graduate
students, humanities faculty, social science faculty, science
faculty.
Target2: A score of at least 3.90 from each of the major
constituencies.
Method: The University Library conducts extensive surveys of
our clientele approximately every other year. A final question
in each survey asks the respondent to "rate your overall
satisfaction…" with the Library using a 1 to 5 scale. This
metric will consider the two latest surveys on record.
Benefits of Scorecard
“…a clear understanding of what drives value
within your area and what doesn’t, greater
insight into senior management’s strategic
plans, and a better knowledge not only of the
strategic role you play within the organization
but how you can enhance that role and sit at the
decision-making table”
Joseph DeFeo, Measuring What Matters.
Industrial Management, v.42,n.3, May 2000
LibQual+(TM)
• Based on ServQual
• Collaboration of Association of Research
Libraries (ARL) & Texas A&M University
Libraries
• Diagnostic tool for measuring library users'
perceptions of service quality
• Identifies gaps between desired, perceived,
& minimum service expectations
• Defined survey questions, dimensions, &
data gathering processes for academic
libraries
http://www.libqual.org/
Benefits of LibQual+(TM)
• Provides better understanding of patrons’
perceptions & desires, & how services are
meeting these
• Libraries using this have learned that they
must keep narrowing categories & questions
Cost-benefit
• Does the benefit exceed the cost?
• St. Louis Public Library
• Measures the value people place on the
consumption of a service in excess of what
they pay to receive that service
• Found benefits received >$10 for each dollar
of tax support
• Other public libraries place a fair market
value on their output measures
• San Diego Library and Miami-Date Public
Libraries; benefits exceed costs by 6:1
Benefits of cost-benefit
• Stakeholders may easily understand
• Difficult, however, to determine valid market
prices
Understand
the
context
Align
objectives
Define
measures
Critical Success
Factor:
Manage
collection
Measurement
Framework
Adapted from Measuring What Matters: A Library/LRC Outcomes
Assessment Manual by Lindauer
Interpret
data
Communicate
results
Context for your framework
•
Who are your key stakeholders?
•
•
How do they prove themselves?
•
•
Who do you really need to prove your value to?
What measures/indicators do they use?
Ask yourself, “in our environment, what is it critical
for us to we measure?”
Page 1 of Framing Template
Framework: Align
• This is where you invest the most time
• Requires conversations with councillors, or
local government “superiors”
• Profile critical stakeholders
• What are their goals & objectives?
• What makes them ‘look good’ to their
superiors or constituents?
Page 2 of Framing Template
Align: Profile Stakeholders
• Stakeholders are those who can put a stake
of support under your organization or a
stake of destruction through your
organization
•
•
•
•
Superiors
Funders
Constituents or market
Clients & potential clients
Align: Know Your Stakeholders
•
•
•
•
•
•
•
•
Who are your stakeholders
How do they determine value?
What comprises value to them?
How do they express it?
What are the most important questions they have?
What are the most important decisions they make?
What are they communicating to their clients?
Which of their goals & objectives do you contribute
towards?
Framework: Define
•
•
•
•
Clearly define this alignment by articulating &
documenting your goals & objectives
Clarify how these contribute towards your
stakeholders’ desired outcomes
Test these goals & objectives with your
stakeholders
Test them with staff to ensure they are in keeping
with your purpose & will aid with planning &
decisions
Page 2 of Framing Template
Framework: Identify
•
•
How will you track your success/progress towards
meeting these goals & objectives?
What indicators or measures will you use?
–
–
–
–
Qualitative? Quantitative?
What data needs to be collected? How?
Keep it key (KISS principle)
Don’t get mired
Page 3 of Framing Template
Framework: Collect
•
•
•
•
•
Do it
Determine a collection schedule
Are you currently collecting data or indicators that
are no longer relevant?
How long do you need to keep data?
Who is responsible?
Page 3 of Framing Template
Framework: Analyze & Interpret
• So…..what?
– What does the data say? What doesn’t it say?
– Examine it from various angles
– What is the progress towards the goals &
objectives?
Page 3 of Framing Template
Framework: Communicate
•
•
•
•
•
Actually begins back at the ‘define’ stage
If your goals & objectives are meaningful for
stakeholders, your measures will be too
Your message to them
If your goals & objectives are meaningful for your
planning & decision-making, your measures will be
indispensable
Your message to you & to staff: “we have to start,
we have to stop & we have to continue…”
Page 3 of Framing Template
Oakville Public Library
• Background
• Oakville Library Board operates on 3 year
business planning model
• Arms-length relationship with Town of
Oakville Community Services Commission,
operates on a rolling 3 year Integrated
Business Planning model
• Brand focused business plan goals and KPIs
Context
• Key stakeholders: Municipality, board,
donors, public, staff, other libraries
• Measures that are important
– efficiency measures (ROI) and
– effectiveness measures (usage, market
penetration etc.)
ALIGN and DEFINE - Oakville Public Library
Business Plan 2005 At A Glance
Vision
“The Centre for Learning – Your Gateway to Knowledge”
The ultimate goal, a
future state, a picture
of what we want to be.
Mission
The purpose of the
organization, for all
stakeholder groups.
Objectives
Measures how well we
achieve our mission.
Strategies
The means used to
achieve the mission.
To provide the ideal environment for
the customer’s learning experience.
•
•
•
•
To support, educate, motivate and recognize
staff and volunteers in providing the ideal
environment for learning.
Increase awareness
Increase value in the community
Increase usage
Align organizational culture with external position in the community
Customer Service Priorities
• Brand Integration
• Collections Management
• Outreach Services
• Early Childhood Literacy
Corporate Priorities
• Strategic Planning
• Board Orientation & Advocacy
• Human Resources Strategies
• Town Integration
• Infrastructure Management
• E-services Development
IDENTIFY – KPIs used at Oakville
Value
Maintain satisfaction level in Citizen’s Survey
Maintain Cardholders per capita 70%
Usage
Number of service transactions per capita
47.56
Awareness
Overall awareness of breadth of programs &
services improved
Align culture
Satisfaction measure through informal
customer satisfaction & staff awareness
survey
Number of
service
transactions
> 1.0 Annual Expenditures
COLLECT
• Tools used for collection include:
– Citizen’s survey – measures value and
awareness every 3 years
– Cardholders per capita (Dynix/Horizon)
– Informal satisfaction surveys and feedback forms
– focus/advisory groups, web surveys, print
surveys
– Collection satisfaction survey
ANALYZE AND INTERPRET
• Service transactions = circulation + people
entering + program attendance + outreach +
information queries answered + etransactions
• Trends analysis
• Comparative analysis with other areas
(libraries, town departments)
• Variance reporting
COMMUNICATION
• Look at frequency, depth and message and
customize for specific stakeholder
• Yearly communication to municipality through
budget process
• Yearly communication to Library board, with
quarterly updates throughout the year
• Annual Report to the Community to donors and
public
• Semi-annual updates to staff through semi-annual
reports, intranet and staff newsletter
Gaps and Hurdles
•
•
•
•
•
•
•
•
•
Consistency in measurement over the years
Ability to measure
Staff engagement and education
Detail of measurement required for different
stakeholders
Meaningful communication
Extrapolation errors – it’s part of the equation
Cause and effect – direct or indirect?
Advocacy plan
Identify comparators across libraries, local
community groups and other non profits etc.
Can you really measure impact?
• Are there times you cannot?
Can we measure impact?
Sometimes what counts can’t be counted, and
what can be counted doesn’t count.
– Albert Einstein
We should be a bit wary of the “little library”
…For when it is good, it is very, very good
and when it is bad, it’s a “pretty good
library for a town this size.”
- E. Rodger, New Zealand Libraries, March 1990
Measuring Impact - “Libraries change lives”
• “Knowledge of alphabet letters at entry into
kindergarten is a strong predictor of reading
ability in 10th grade” – Every Child Ready to
Read, West Bloomfield Township
• “…it helped be improve my ability to help
students in their essay and civic topics; all
useful” says a teacher-librarian of the Youth
Online! program