A Balanced Scorecard is a Process Not
Numbers
MID ATLANTIC EDUCAUSE 2005
Saint Michael’s College
Bill Anderson – Chief Information Officer
Billie Miles – Director of IT
© Bill Anderson and Billie Miles, 2005. This work is the intellectual property of the authors. Permission is granted for this
material to be shared fro non-commercial, educational purposes, provided that this copyright statement appears on the
reproduced materials and notice is given that copying is by permission of the author. To disseminate otherwise or to
republish requires written permission from the authors
.
Here we go…

CIO gained support by offering IT as the BSC pioneers. This
support resulted in funding that allowed us to work with outside
resources

Partnership with Datatel – Donna Taylor and Ken Rice from the
Center for Excellence

Kathie Gilbert – Special Assistant for Quality Initiatives - WNMU
was our Quality Stewart

Bill and Billie - IT leadership committed to the project

IT staff – eager and willing
Saint Michael's College - Mid-Atlantic Educause 2005
2
The steps
1.
Project Plan developed by the CIO, IT Director, and the consulting
team
2.
First meeting with our quality steward
3.
Second meeting with our quality steward
4.
“Baby” steps
Saint Michael's College - Mid-Atlantic Educause 2005
3
Step 1 - The project plan
The plan detailed:

Anticipated benefits

Target date for implementation (July 2003)

Roles and responsibilities

Action plan – included estimated times and delivery
methods
Saint Michael's College - Mid-Atlantic Educause 2005
4
Step 2 - First meeting with the quality
steward
The action plan called for collecting feedback from the constituents served by IT. The
goal was to clarify perceptions and to collect base information.
Preparation

Our quality steward worked with us to create a customer satisfaction survey and
we scheduled focus group meetings for the first day of the visit.
Activities

Day one – focus groups and discussion of IT “history”

Day two – CQI and BSC intros for IT staff

Day three – CIO, IT Director and Quality Steward plan next steps
Outcomes from the visit

Survey analysis

Focus groups reports
Saint Michael's College - Mid-Atlantic Educause 2005
5
Step 3 – Second meeting with the quality
steward
Preparation

Staff discussion of survey and focus group results

Work with IT staff to generate Mission/vision statements, and we did an IT slogan
exercise (We’re IT)
Activities

Day one – more BSC training, measurement, strategy exercises, one on one
Interviews with IT staff

Day two - scorecard design, customer service, more interviews, summarized
workshop materials and met with the President (30 minutes)

Day Three – Debrief with CIO and IT Director – lots of work left to do!
Outcomes of this visit

Report

Selected short term goals and measurements
Saint Michael's College - Mid-Atlantic Educause 2005
6
Step 4 – “Baby” steps
Next steps identified and assigned to the CIO and IT Director
1.
Complete two staff development trainings
2.
Begin regular evaluation and performance measurement
3.
Develop Helpdesk measures for the BSC
4.
Make IT work more visible on campus
Long term goal – develop the permanent scorecard, make data collection,
review and continuous improvement part of the culture
Saint Michael's College - Mid-Atlantic Educause 2005
7
The scorecard
The four quadrants:
1.
Customer measures – goal: Improve customer satisfaction scores
2.
Financial measures – goal: improve operational effectiveness
3.
Internal measures (service delivery) – goal: improve operational
efficiency
4.
Learning and growth measures – goal: increase staff and end user
training opportunities
Each work group (there are 5) is developing measures in these four
areas. These will roll-up into a department scorecard.
Saint Michael's College - Mid-Atlantic Educause 2005
8
Original Goals – 4/2003




1.
2.
3.
Focus on Service Quality
Technology in Teaching
Align day-to-day work with V-2010
Top 3 strategic objectives & perspectives selected
Improve customer satisfaction
Improve Operational Efficiency
Provide User Education & IT Professional
Development
Saint Michael's College - Mid-Atlantic Educause 2005
9
Error Messages During Online Registration
Connection Reset by Peer
Broken Pipe
Internal Error: exception thrown
Error Reading Incoming DMI transaction
Maximum number of sessions reached. Please try again
shortly.
Total Number of Calls
Human Resources
Student Accounts
Students - KnightVision Calls
Admissions
Res Life
Faculty - KnightVision
Faculty - Other
Finance
I.T.
Other
Training for Apps Team Number of Sessions/Attendees
Webinars
Training Classes
Conferences
04/07/2004
0
22
0
91
226
May (10-28)
58
9
9
9
5
4
2
0
5
1
14
04/05/2004 04/01/2004
0
0
16
17
0
0
39
63
151
78
June
90
6
7
1
9
5
0
8
10
11
33
11/10/2003
130
19
200
11
5
July
111
6
10
9
7
1
5
9
17
8
39
August
157
6
7
8
5
3
10
7
14
29
68
0
1/1
1/1
1/4
1/1
0
Saint Michael's College - Mid-Atlantic Educause 2005
11/13/2003
1262
132
258
12
5
10
Instructional Technology - Monthly Summary
This Month
FY 2005
Last Month
Customer Measures
# of Completed Requests for Assistance
Individual software assists
eCollege assists
Classroom technology assists
Specialized lab assists
Software tests
# of Training sessions
Software training (all)
Classroom media training
Workshops offered
29
10
10
7
2
0
35.0
24
10
0
Satisfaction level of workshop participants
Media Setups
Classroom setups
14
Special Events setups
22
Centennial setups
2
Overtime hours
Internal Measures
Use of facilities
LLRC Classroom Reservations
121
BER 105 orientations
0
FacultySaint
Development
sign-ins
23
Michael'sLab
College
- Mid-Atlantic Educause
2005
40
FY 05
AVG
7
9
17
5
4
21.0
15
2
4
35
8.5
9.5
12
3.5
2
28.0
19.5
6
2
9
19
0
11
20
1
93
0
20
107
0
21
11
FY 2005
This Month Last Month
Requests for Software
# of Requests
# of New Software Purchases
Average Time to First Response (hrs)
AverageTime to Close (days)
# of Install Requests
Average Time to First Response (hrs)
AverageTime to Close (days)
# of Cancelled Requests
11
2
20.6
3.7
9
9.6
16.3
0
30
2
10.9
20.5
28
9.6
3.1
1
FY 2005
This Month Last Month
LRP Warranty Service Calls
# of Service Calls
# of Service Calls Placed for Units in 1st Yr
# of Service Calls Placed for Units in 2nd Yr
# of Service Calls Placed for Units in 3rd Yr
LRP Allocation Changes
# of PC Allocation Changes
New Placements
Model Swaps
Sabattical Loaners
Temporary Loaners
Reallocations
FY 05
AVG
15.0
2.0
10.6
10.1
13.0
7.3
8.4
0.3
3
1
2
0
FY 05
AVG
3.7
0.3
3
0.3
3
3
0
0
0
0
FY 05
AVG
26.7
18.7
0.0
0.7
1.3
6.0
3
0
3
0
FY 2005
This Month Last Month
66
53
0
1
3
9
Customer Satisfaction Measures
FY 2004
This Month Last Month FY 04
AVG
41
18
21.2
3
0
3.2
0.0
0.0
25.0
5.3
0.0
18.8
38
18
18.0
5.6
11.8
7.4
6.9
5.9
6.5
0
0
5.0
Customer Satisfaction Survey
FY04
Importance Satisfaction
54
21
Internal Operations Data
FY 2004
This Month Last Month FY 04
AVG
n/a
n/a
6.0
n/a
n/a
5.5
n/a
n/a
0.2
n/a
n/a
0.3
FY 2004
This Month Last Month FY 04
AVG
7
4
3.7
3
2
2.5
2
2
0.6
0
0
0.2
2
0
0.4
n/a
n/a
n/a
Saint Michael's College - Mid-Atlantic Educause 2005
Gap
33
FY 03
Importance Satisfaction
29
28
Gap
1
Targets
Reduce Gap by 50%
Satisfaction Rating of 80%
Reduce Time to Close by 20%
Reduce Time to First Response by 20%
Target: Reduce Service Calls by 50%
12
Goal - Monitor contract trends / new vendors
Cost per minute
- 2004 Last
This
Average
Month
Month
Month
Month
Target #
To Target
Trend
0.03970
0.05713
0.04362
0.03970
0.05710
0.04364
0.03724
0.05675
0.05654
0.03721
0.05675
0.05591
0.0289
0.0523
0.0388
0.0108
0.0048
0.0048
#Calls Change
Over
Over
Over
Trend
5371
624
2393
Increase %
Increase
Increase
Local Calls
Intrastate Calls
Interstate Calls
Call Volumes
27054
3373
8219
Local Calls
Intrastate Calls
Interstate Calls
Overall Effectiveness:
Contract Negotiations / other savings
Work orders
Phone Changes
Repairs
23029
3133
5520
YTD
$29,850.00 $10,000.00
Customer Satisfaction: Service Volumes
Goal - Analysis / Followup
- 2003 This
Last
YTD
Operational Effectiveness: Cost Management
Total
Start 6/04
621
na
- 2004 Last
Month
380
na
28969
3537
8492
25877
3196
6154
23598
2913
6099
This Month
$0.00
This
Month
21
37
Saint Michael's College - Mid-Atlantic Educause 2005
13
Sep-04
2004
491
290
Number of calls PLACED
Number of calls CLOSED
2003
771
na
Percentage of calls for SOFTWARE
Percentage of calls for HARDWARE
90%
10%
94%
6%
Percentage of calls Faculty/Staff
Percentage of calls Student
62%
38%
72%
28%
1 Hour
Time to Close Calls (Actual) as a Percentage
This Month
1 Month Ago
2 Month Ago
3 Month Ago
4 Hours
22%
34%
25%
31%
4%
9%
12%
11%
8 Hours
24 Hours 48 Hours 72 Hours >72 Hours
1%
5%
4%
4%
6%
8%
11%
10%
7%
11%
12%
14%
4%
4%
10%
5%
56%
30%
26%
25%
Services
Profile & Home Directory
Print Service
Application Service
Email Service
Internet Service
Mikenet Service
Spam Appliance
Barracuda
Spam
860497
69%
Virus
Total In
44312 1252409
4%
Saint Michael's College - Mid-Atlantic Educause 2005
14
Modified Goals – 6/2004

Results of 2004 Customer Satisfaction
Survey - we added improvement picks




Responsiveness to request for new software
Usefulness of responses
Web mail
Use and support of classroom technology
Saint Michael's College - Mid-Atlantic Educause 2005
15
Departmental Level Scorecard
First pass in December 2004
 Work in progress
 Evolving toward:

 More
customer feedback measures
 Higher level of abstraction ( letter grades?)
 Need to integrate this into the way leaders do
business
 External benchmarks
Saint Michael's College - Mid-Atlantic Educause 2005
16
IT - Combined BSC - November 2004 Data
Learning & Growth Measures
IT Staff - Professional Development
Applications
ITC
Saint Michael's College - Mid-Atlantic Educause 2005
Nov Oct
0
1
2
1
17
IT - Combined BSC - November 2004 Data
Financial Measures
Instructional Technology Group
$$ Loss due to theft
Nov Oct
Resource Mangement
$$ Loss due to theft
Telecommunications
Saint Michael's College - Mid-Atlantic Educause 2005
18
IT - Combined BSC - November 2004 Data
Internal Measures
Instructional Technology Group
Use of FDL
Resource Mangement
Service calls for Y-1 units
Service calls for Y-2 units
Service calls for Y-3 units
Nov Oct
32
37
3
3
0
0
1
0
YTD Fail
1.3%
1.9%
0.7%
Telecommunications
Saint Michael's College - Mid-Atlantic Educause 2005
19
IT - Combined BSC - November 2004 Data
Cu s to mer Mea s u res
Applications Team
Number of calls
Nov
Oct
133
129
Instructional Technology Group
Number of classroom technology assists
33
18
Resource Mangement
Avg # days close sftwr requests
12.2
Telecommunications
% of students checking voicemail
96%
User Services
Number of calls placed
Number of calls closed
% of calls closed 1 to 8 hours
% of calls closed 24 to 48 hours
255
344
46%
9%
156
373
8%
6%
Network Services
Email (Shadow) - uptime
Web Mail (SMCMAIL2) - uptime
Knightvision (Spirit) - Total visits
Number of service interruptions
Number of alerts
96%
96%
14,050
8
10
3,874
7
3
Online registration errors
0.8
4-Nov Nov-03
62
2,034
Saint Michael's College - Mid-Atlantic Educause 2005
20
Saint Michael's College - Mid-Atlantic Educause 2005
21
Building on what we’ve got
Another level of detail
Accuracy of “data entry”
Add 1 measure – address usefulness
Saint Michael's College - Mid-Atlantic Educause 2005
22
Lessons So Far

Change is hard

Unplanned pushes out the planned

We have met the enemy and he is us

Ready Fire Aim - It’s better to do something and adjust
than to fail to act for lack of the perfect plan.

Must connect the dots

Change the IT focus – customers not gadgets
Saint Michael's College - Mid-Atlantic Educause 2005
23
What’s next?

CIO and Director - measures versus process

Quality steward is going to visit in April 2005 - Our 2 year
anniversary

Our “official” scorecard presentations began in January 2004

Training – maybe more effective now that we have some ‘hands-on’
experience.

Make better use of the helpdesk software – we are going to relaunch the product. Use it as a department tool, rather than just for
the Helpdesk.
Saint Michael's College - Mid-Atlantic Educause 2005
24