Outcomes Evaluation
in Academic Law Libraries
Outline/Agenda
What is it?
Why should you care?
How do you do it?
Other questions?
What is “Good?”
How Much?
How Well?
How Many?
How Valuable?
How Economical?
How Reliable?
How Prompt?
How Courteous?
How Accurate?
How Satisfied?
How Responsive?
Types of Measures
Inputs:
Outputs:
Service Quality:
Satisfaction:
Outcomes:
resources that a library
dedicates to a program
use of library resources
“gap model”
overall emotional and
personal reaction to
program
changes that resulted from
the program; “impact” of
program
Q: How do you determine the value of an
academic law library?
A: Size Matters!
Recent Trends in Academia
1998 Amendments to Higher Education Act of
1965, Pub. L. 105-244, 112 Stat. 1581
Regional Accrediting Bodies
Funding agencies
Professional associations
Change on the horizon?
Calls for changes in legal education
– ABA Task Force on Law Schools and the Profession,
MacCrate Report (1989)
– Gregory Munro’s Outcomes Assessment in Law
Schools (2001)
Regional Accrediting Bodies
Recognition process
– Proposed Standards for Approval of Law Schools
The Bigger Picture . . .
No one believes volume counts and budgets are
adequate measures
No one can afford to keep growing
Leads to improved services
Good tool for resource allocation
Allows us to tell the library’s story . . .
Overview of
Outcomes Evaluation Process
Step One:
Choose a program to evaluate
Step Two:
Decide how to collect data
Step Three:
Collect the data
Step Four:
Analyze and interpret results
Step Five:
Use the results
Step One: Choose a Program to Evaluate
Focus on a single program/service
Write down the context surrounding the
program
– Inputs?
– Outputs?
– Patrons?
– Desired outcomes?
Step Two: Decide How to Collect Data
Best Method: situational, but usually a
combination of several methods
Questions:
– What are you trying to measure?
– For what purpose will the data be used?
– What are your resources?
(money, staff, time frame, etc.)
Some Typical Data Collection Methods
Pre- and Post-Testing
Portfolios
Observation Techniques
Surveys and Questionnaires
Interviews
Focus Groups
Step Three: Collect the Data
Some issues:
– Who will collect the data?
– When to collect the data?
– How many responses needed? Sampling?
Pre-test your instruments
Keep staff informed throughout the
process!!
Step Four: Analyze and Interpret Results
Qualitative data analysis:
– Get to know the data
– Locate patterns, themes, sub-themes
– Substantiate with evidence
– Replicate
– Optional: code data for use in statistics software
program
Step Four (cont.)
Quantitative Data Analysis: All you need is
a pencil and paper!
– Simple means and medians can be enough
– Consult with faculty, local charity groups, etc.
– Commercial statistics programs
Step Four (cont.)
Organize data into charts, reports
Compare results with desired outcomes
– Any positive outcomes that you did not anticipate?
– Any negative outcomes that you did not desire?
– Did you fail to find evidence of the desired positive
outcomes?
– Any explanations available?
Step Five: Use the Results
VERY important step!
Reports, charts are great communication tools
Make Improvements
– Any aspects of programs that don’t measure up?
– Any aspects that can be tweaked, changed, or
eliminated?
– Any positive outcomes that you did not expect but that
you want to foster in the future?
Step Five (cont.)
Resource allocation
– Are additional resources needed to improve
the service?
– Is the program worth the resources devoted to
it?
– Can you collaborate with other interested
parties to make the service better?
Marketing: Publish your positive results!
Final Thoughts
Outcomes are only part of the story
Start small and keep it simple
“Think positive about the negatives”
An academic law library may not be heaven, but we all
should strive to have deliriously happy patrons!
Questions?