THE ‘UNDERSTANDING LIBRARY
IMPACTS’ PROTOCOL:
DEMONSTRATING ACADEMIC LIBRARY CONTRIBUTIONS TO
STUDENT LEARNING OUTCOMES IN THE AGE OF ACCOUNTABILITY
Derek Rodriguez
Doctoral Candidate, School of Information and Library Science
The University of North Carolina at Chapel Hill
Program Officer, Triangle Research Libraries Network
9th Northumbria International Conference on Performance Measurement in Libraries and
Information Services, August 23, 2011, York, England
Undergraduate education in the age of
accountability
Concerns
Metrics
• Access and cost
• Retention
• Attainment
• Graduation rates
• Competencies of
• Student learning
graduates
outcomes
Student learning outcomes
• Locally defined learning outcomes
• General education and the academic major
• Broad abilities and cross-discipline skills
• e.g. Critical thinking, inquiry and analysis, information
literacy, quantitative literacy, written communication
• VALUE rubrics (Rhodes, 2010)
• Discipline specific knowledge and skills
• By academic major, e.g. History, Chemistry, Nursing,
etc.
• Tuning projects (Gonzalez & Wagenaar, 2005; ICHE,
2010)
- Rhodes, T., ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC:
Association of American Colleges and Universities.
- González, J. and Wagenaar, R. (eds.) (2005). Tuning Educational Structures in Europe II. Bilbao, ES: University of Deusto.
- Indiana Commission for Higher Education. Tuning USA Final Report: The 2009 Indiana Pilot, 2010.
The challenge for libraries
- Connecting library use to student learning outcomes
Library
use
?
Student
effort
Student
Learning
Assessment
Program
reviews
External
frameworks
VALUE & Tuning
‘Understanding Library Impacts’
- Explores library use as a component of student effort
Library
use
Learning
Activities
Student effort
Student
Learning
Assessment
Program
reviews
External
frameworks
VALUE & Tuning
A focus for library assessment
Capstone
experience
Academic
major
General
education
Design for pilot study (spring 2011)
• Population: undergraduate history majors enrolled at 2
•
•
•
•
sites, a liberal arts college and a liberal arts university in
the U.S. (N=74)
Project: 20 – 40 page research paper using evidence from
primary sources
Step 1: Constructed a learning activities crosswalk
using syllabi and rubrics for research papers
Step 2: Students completed a web-based critical
incident survey after turning in papers
Step 3: Reports were distributed to study sites via
secure, database-driven web-sites
Understanding Library Impacts
- Explores library use as a component of student effort
Learning
Activities
Library
use
Student
Learning
‘Capstone’
project
Critical
incident
survey
Learning
activities
crosswalk
Assessment
Program
reviews
External
frameworks
VALUE & Tuning
Learning activities crosswalk
Activities
Getting oriented
Capstone-related outcomes
Discipline-specific skills
-
Locates secondary and primary sources
Distinguishes among types of sources
Choosing a topic
Thesis and argument
Developing a
thesis
Gathering
evidence
-
Develops original thesis statement
Advances argument in support of thesis using
evidence from primary sources
Evidence and analysis
Finding other
sources
-
Creating a
bibliography
Writing and Citing
-
Writing
-
Evaluates and interpret primary sources
Uses secondary sources to provide context
Demonstrates an understanding of the methods
of history
Communicates argument in a coherent, wellwritten paper
Follows disciplinary style and citation standards
Critical incident survey
- Developed and refined in qualitative studies (2006, 2007)
Deliverables
and outcomes
Learning
activities
Open
ended
questions
and probes
Helps and
problems
Library use
- Electronic
resources
Top-ranked
e-resource
- Traditional
resources
Top-ranked
traditional
resource
- Services
Top-ranked
service
- Facilities /
Equipment
Top-ranked
facility
Local
questions
Demographics
and affect
Academic
challenge
Most common library uses, n=41 (55%)
100%
library catalog
books
electronic journals
printers
78%
study space
internet search engines
73%
reference
indexes or databases
computers
68%
interlibrary loan
non-library web-site
electronic books
63%
digitized primary sources
library instruction
reference books
manuscripts or archival materials
41%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
‘Top ranked’ use types by activity (n=41)
100%
93%
83%
% of respondents
80%
60%
40%
20%
0%
51%
56%
Electronic resources
Traditional resources
Services
Facilities/Equipment
‘Top ranked’ resource use while 'gathering
evidence'
manuscripts or archival materials
books
library catalog
non-library web sites
research guide
electronic journals
indexes or databases
electronic books
digitized primary sources
Site A
Site B
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
% of respondents
Outcomes related to the activity ‘Gathering evidence’
-
-
Locates secondary and primary
sources
Distinguishes among types of sources
-
Advances argument in support of thesis
using evidence from primary sources
-
Evaluates and interprets primary sources
‘Top-ranked’ services and facilities used
during 'writing'
interlibrary loan
research consultation
library instruction
study carrel / workspace
Site A
Site B
printers
group study room
computers
0%
10%
20%
30%
% of respondents
40%
50%
Outcomes related to the activity ‘Writing’
-
Communicates argument in a coherent,
well-written paper
Follows disciplinary style and citation
standards
-
Advances argument in support of thesis
using evidence from primary sources
-
Uses secondary sources to provide context
Factors of use (helps and problems)
• Derived from qualitative studies and LIS literature
• Gathered for ‘ranked uses’ only
• Categorized for analysis
• Example category: ‘Help finding information’
• “I learned about information sources for my project” (help)
• “I learned new skills” (help)
• “helped me when I got stuck” (help)
• “the assistance I received wasn't helpful” (problem)
• “it was difficult to find someone to help me” (problem)
Proportion of students reporting factors of use during
capstone project, by theme, n=41
Access to information
Convenience
Availability of space
Theme
Ease of use
Help finding information
Helps
Problems
Affect of staff
Access to tools
Anxiety
Issue with assignment
Gaining independence
0%
20%
40%
60%
% of respondents
80%
100%
Challenges faced during the project
Assignment
5%
18%
47%
29%
6%
Site A, n=17
Evaluating
information
Finding
information
Time
management
36%
45%
14%
Site B, n=22
“Overwhelmed by various themes and layers of my research” (A-2)
“Putting all of my information together into a cohesive paper” (A-5)
“Time management” (A-8)
“Several of the books I needed were being rebound…” (B-3)
“Uncertainty about where to start” (B-10)
“I was worried about finding primary source material” (B-15)
“At a somewhat early point in the project, I realized that the scope of
my project was much too wide …” (B-14)
Most important library uses, by type of use, n=37
38%
35%
E-Resources
Traditional resources
19%
Facility
8%
Service
“… those databases, JSTOR specifically, really helped me.” (B-4)
“Library space to work on my thesis both writing and research” (B-2)
“being able to get books from off campus” (B-3)
Alternate activity, if first choice hadn’t been available, n=29
10% 4%
14%
52%
17%
3%
Struggle
Change topic
Get by
Seek help
Purchase materials
Visit other libraries
Understanding Library Impacts protocol
- Contributions and implications
Library
use
Learning
Activities
Student
Learning
‘Capstone’
project
- Credible connections between library
use and expectations for student learning
- Generates rich data useful for advocacy
and improvement
- Places library in important campus
conversations about student learning
Assessment
Program
reviews
External
frameworks
VALUE & Tuning
Next steps
Refining instruments; conducting reliability and validity testing
Exploring “patterns and predictions”
Continue evaluation with history majors at 4 more sites (fall 2011)
Future work
Evaluate protocol with other disciplines and settings
Explore integration with assessment systems and analytics projects
Questions?
Contact: [email protected]
Crosswalk: Broad abilities and disciplinespecific outcomes
• VALUE Rubrics (Rhodes, 2010)
• Critical thinking
• Inquiry and analysis
• Information literacy
• Written Communication
• Tuning Outcomes for History (ICHE, 2010)
• Historical knowledge
• Thinking and analytical skills
• Communication skills
• Personal motivation
Rhodes, T., ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington,
DC: Association of American Colleges and Universities.
Indiana Commission for Higher Education. Tuning USA Final Report: The 2009 Indiana Pilot, 2010.
Top ranked resource use while 'gathering
evidence'
manuscripts or archival materials
books
library catalog
non-library web sites
research guide
electronic journals
indexes or databases
electronic books
digitized primary sources
Site A
Site B
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
% of students
Related capstone outcomes
-
-
Locates secondary and primary
sources
Distinguishes among types of sources
-
Advances argument in support of thesis
using evidence from primary sources
-
Evaluates and interprets primary sources
Outcomes related to ‘gathering evidence’
Activity
Gathering
evidence
Project outcome
Advances
argument in
support of thesis
using evidence
from primary
sources
VALUE
Inquiry and Analysis
Analysis - Organizes
and synthesizes
evidence to reveal
insightful patterns,
differences, or
similarities.
Tuning Outcome
Site A
Site B
Formulate and test
plausible historical
hypotheses and
marshal an
argument.