Getting to Use:
What Stimulates
and Impedes
Use of Student
Engagement
Results?
Jillian Kinzie, Alexander
McCormick, Danny Olsen,
Charles Blaich, Kathleen Wise
AIR Forum, Denver -- May 27, 2015
Session Goals
 Gain familiarity with field-tested
insights into the obstacles and
strategies for using data
 Give input on what stimulates and
impedes data use
 Consider the role of IR in furthering
data use
 Contribute to building a repository
of field-tested practice in data use
 Q&A
Moving from Data to Action
NSSE is more than a
survey: It’s an
agenda for evidencebased improvement
Collecting data is the
easy part!
This session aims to
unpack next steps
Identify
goals
Evaluate
impact
Gather
data
Take
action
Analyze
results
Interpret
results
Poll Questions…
 Does your campus prepare tailored NSSE
reports (beyond what NSSE sends)?
 Is there a person/committee responsible for
acting on your NSSE results?
 What proportion of your IR and assessment
time is spent on data use?
a)
b)
c)
d)
None
Up to a quarter
About half
More than half
Applying,
translating,
putting results to
work
Reflection Questions…
 What is your role in promoting data use?
 How often to you meet with departments/
programs to find out what they'd like to know
about their students?
What’s going on in the field?
Systematic review of
examples of NSSE use:
 How are institutions
using data?
 What motivated
data use?
 What approaches are
employed to make data
actionable?
 What resulted from
data use?
Field-Tested Insights for
Fostering Data Use
1. Enlist faculty, staff, & students in planning
data use from the beginning.
2. Connect NSSE to specific issues of concern.
3. Create a plan of action that outlines what
you hope to do with your data.
4. Form questions about target areas of inquiry
before you receive results.
5. Examine results alongside other data.
6. Take advantage of all possible motivators.
7. Identify data-use “champions.”
OUR PANEL
Danny Olsen, Brigham Young University
Charles Blaich, Center of Inquiry, Wabash
College, and HEDS Consortium
Kathleen Wise, Center of Inquiry, Wabash
College, and HEDS Consortium
NSSE at BYU
 Increased awareness and utilization with
evolution of campus culture of assessment
 Data utilized at institutional, college,
department, school, and program levels
 Triangulation of resulting data with other
institutional data
 Enhanced interest and usage of NSSE data
(seniors) with the creation of program-level
reports
Meaningful Comparisons?
DILBERT © 2014 Scott Adams. Used By permission of UNIVERSAL UCLICK. All rights reserved.
BYU Program Level NSSE Report
BYU Program Level NSSE Report
BYU Program Level NSSE Report
BYU Program Level NSSE Report
BYU Program Level NSSE Report
BYU NSSE Uses
 Strategic Assessment
 “Used NSSE data to recalibrate our approach to Freshmen in
Honors, and in undergraduate education”
 “Among other factors, the poor scores on our NSSE survey
results were used to justify reducing class sizes to allow more
faculty/student interaction”
 “We have used NSSE data to help us think about experiences
we want our students to have—especially in off-campus realworld experiences.”
 “After reflecting on the data, we decided to place greater
emphasis on students’ use of technology in courses offered and
in degree requirements.”
 Program Assessment and Review
 “We used the NSSE report for our program assessment.”
 “I use the reports in our unit review.”
BYU NSSE Uses
 Curriculum/Pedagogy
 “Exploring a quantitative reasoning course on campus”
 “We require students to write a short paper before they
graduate outlining connections between what they have learned
as part of their degree and future career/educational plans”
 Learning Outcomes
 “We’ve used the data to report ways our students have met
learning outcomes.”
 Resource Planning
 “We hired a new FTE and became a limited enrollment program
to reduce section sizes successfully”
 Accreditation (Specialized & Regional)
 “I carefully review the report and combine the data with what I
get from my exit interviews with graduating seniors and then
this forms a part of our annual department retreat discussion,
which we use as part of our accreditation process.”
What Stimulates and Impedes Use of Student Engagement Results?
Of Assessment Data in General?
www.centerofinquiry.org
www.hedsconsortium.org
What impedes it?
• Relying on “magic bullet” reports
- Reports to nowhere
o Distributed via email with no occasion for conversation
o
Posted on the password‐protected IR/assessment website in the hope that they will be downloaded
o
“Boil the ocean” reports with lots of data, lots of jargon, overly complicated graphs, and no focus
www.centerofinquiry.org
www.hedsconsortium.org
What impedes it?
• Relying solely on the power of evidence
- Emotion
o Perceptions of risk will guide people’s response to the data
- Motivated reasoning
o The tendency to bias our evaluation of information to meet some goal - Confirmation bias
o To search for, interpret, or recall information in a way that confirms one's beliefs or hypotheses
www.centerofinquiry.org
www.hedsconsortium.org
In effect, individuals are cognitively motivated to reject information about risk when they perceive that accepting it would threaten their defining group commitments. To avoid this reaction, then, information about risks must be framed in a way that affirms rather than denigrates recipients’ cultural identities; to make it possible for persons of diverse cultural persuasions to experience that affirmation simultaneously—
and thus reach consensus on a contested risk issue—the information must be framed in a way that expresses a plurality of social meanings. (page 765)
When societal risks become suffused with antagonistic social meanings, it is (often if not always, and with respect to many if not all issues) individually rational for ordinary members of the public to attend to information in a manner that reliably connects them to the positions that predominate in their identity‐defining groups. (page 420)
www.centerofinquiry.org
www.hedsconsortium.org
Implications
• “Do it to ourselves before they do it to us” or “the accreditors are coming!!” do more harm than good
• Tie assessment evidence, both in presentation and action, to staff and faculty values
-
Inquiry
Advocacy for students
Support faculty and staff efforts to teach well
Departmental, programmatic, and disciplinary tribes
• Reduce cognitive load
- Create time and space for deeper conversations and sense making
• Develop relationships with staff and faculty
www.centerofinquiry.org
www.hedsconsortium.org
What stimulates it?
• Letting people discover data rather than reporting it to them
- Creating time and space for people to review and make sense of evidence in the presence of colleagues
- Bringing “unusual combinations” of people together to consider and identify important points in the data • Creating a curriculum
- Developmental sequence of reports, structures, and activities that support momentum towards change
- Getting resources and visible support from senior leadership that these data matter
- It’s a long‐term process
www.centerofinquiry.org
www.hedsconsortium.org
What stimulates it?
• Thinking of and talking about assessment as a faculty and staff development project
- Assessment is evidence‐informed faculty and staff development
- Linking to other, ongoing staff and faculty development programs
• Working from an established relationship
- What are your faculty and staff interested in about their students?
- What are they afraid of?
- Where are their stress points? www.centerofinquiry.org
www.hedsconsortium.org
“Why did you listen to her?” I asked. “She had only a fraction of your experience.”
In the beginning, she didn’t, the nurse admitted. “The first day she came, I felt the workload on my head was increasing.” From the second time, however, the nurse began feeling better about the visits. She even began looking forward to them.
“Why?” I asked.
All the nurse could think to say was “She was nice.”
“She was nice?”
“She smiled a lot.”
“That was it?”
“It wasn’t like talking to someone who was trying to find mistakes,” she said. “It was like talking to a friend.” www.centerofinquiry.org
www.hedsconsortium.org
What stimulates it?
• Connecting students – engaging student voice
-
Using the student comments from NSSE
Student focus groups
Student‐led focus groups
Having students participate in sense‐making meetings about data
www.centerofinquiry.org
www.hedsconsortium.org
LET’S APPLY WHAT WE’VE
LEARNED…
Updated NSSE – New Opportunities
for Data Use
NSSE Data Scenarios
1. Disappointing Results. 3 years
consistently low scores on the
Student-Faculty Interaction EI.
 What are your 1st steps upon receiving
results? With whom do you share
findings? What approaches might you
take to explore scores? What do you
anticipate from campus audiences &
how might you address to ensure data
are thoughtfully considered?
NSSE Data Scenarios
2. Data Hotspots. 43% response rate for NSSE 2014;
eagerly anticipated NSSE results for accreditation reaffirmation
& retention task force. But, concerning results: FY scores on
the Effective Teaching Practices EI and Discussions with
Diverse Others & diversity experiences are lower than all comp
groups; measures for SR look good with slightly below comp
groups on Higher Order and Reflective & Integrative Learning.
Your faculty believe strongly in their teaching effectiveness
and diversity has been a bit of a hotspot for the campus. You
expect campus audiences to be concerned and perhaps
surprised by these results.
 What is your 1st step? What approaches might you
take to ensure productive use of results?
NSSE Data Scenarios
3. Ho-Hum Results. Your campus’s NSSE
results are on par with your customized peer
group for just about every Engagement
Indicator (EI); 4 of 10 EIs are higher than
the broader Carnegie group (but the effect
size is small).
 What approaches might you take to
generate attention to these solid and
generally agreeable results?
Advancing Good Data Use Practices
 Share your data use stories.
 What data use hurdles demand
more attention from the field?
 What can NSSE do to help advance
your data use?
New Resource: NSSE Data User’s Guide
Simply reporting NSSE results
will not lead to action
 Designed to help plan and facilitate
data-driven conversation using
NSSE and FSSE data
 Helps you get organized and
provides suggestions for leading
a workshop or session
 Six worksheets with guided exercises
 Offers tips for overcoming obstacles
when sharing NSSE and FSSE data
 New webinar available:
nsse.indiana.edu/html/webinars.cfm
nsse.indiana.edu/html/data
_and_results_guide.cfm
Registration Open for NSSE 2016
 Invitation for 2016
 Deadline to register:
September 25, 2015
 Register online:
nsse.iub.edu