Creating Valid And Reliable Reactionnaires That
Predict Behavior Transfer
Mary L. Lanigan, Ph.D.
Introduction
This session will enhance your knowledge and skills in creating valid
and reliable reactionnaires that can predict behavior transfer. These
evaluation skills are important to learn because many practitioners use
reactionnaire data as the only measure of training success. Even with its
popularity, practitioners create instruments that are biased towards the
positive, collect general information, lack content validity and are
unreliable. This session shows you how to overcome these problems
and create sophisticated evaluation tools.
Objectives
At the end of this training session, you will be able to:
 edit corporate reactionnaires to include six assessment guidelines;
 create a self-efficacy section within a reactionnaire;
 use the self-efficacy section to prove content validity and internal
consistency reliability: and,
 predict behavior transfer using post-training self-efficacy scores.
Outline
To meet the above objectives, the session outline is:
I. Edit corporate reactionnaires to include six assessment guidelines.
a. Practice
b. Debrief
II. Create a self-efficacy section within a reactionnaire.
a. Practice
b. Debrief
III. Use the self-efficacy section to prove content validity and internal
consistency reliability.
IV. Predict behavior transfer using post-training self-efficacy scores.
V. Final questions
Mary L. Lanigan is a human performance professor at Governors State University, and is the author of evaluation
and technology books. She has written articles and presented at national conferences on evaluation and other
human performance topics. She is the recipient of two ISPI Awards of Excellence-- one for her evaluation book
and another for her evaluation research. She has also received a Governors State Faculty Excellence award. Her
Ph.D. is from Indiana University’s Instructional Systems Technology program and she possesses three Masters
degrees in Educational Psychology, Communication, and Instructional Systems Technology. Mailing address:
Governors State University, CAS, University Park, Illinois 60466. E-mail: [email protected].
Copyright  2002 Mary L. Lanigan, Ph.D.
1 of 10
I. Edit corporate reactionnaires to include six assessment guidelines.
Copyright  2002 Mary L. Lanigan, Ph.D.
2 of 10
I. Continued -- Edit corporate reactionnaires to include six assessment guidelines.
Copyright  2002 Mary L. Lanigan, Ph.D.
3 of 10
I. Edit corporate reactionnaires to include six assessment guidelines.
Practice: For each reactionnaire item, indicate which guideline is being violated.
Program Evaluation
Directions: Circle the response that represents your belief regarding the following
statements.
Comments Regarding Each Statement
1.
My expectations about the training
content were:
Surpassed
2.
Met
The training session has increased my
ability to perform my job.
Agree
3.
Disagree
Disagree
Disagree
Okay
Disagree
Enough
Not Enough
The instructor was organized and prepared.
Agree
9.
Not Sure
The amount of time provided for questions
was:
Too Much
8.
Useless
The training materials were presented in
a logical sequence.
Agree
7.
Not Sure
The training games, exercises, etc. were:
Useful
6.
Not Sure
I possessed the knowledge/skills needed to
attend this program.
Agree
5.
Not Sure
The practices and quizzes offered during training
helped me understand the training content.
Agree
4.
Beneath
Disagree
Not Sure
What did you like best about this training session?
10.
What did you like least about this training session?
Demographic Information
Gender: ___ Male
___ Female
Years at Company: __
My present position matches the intended trainee
description listed in the personnel handbook.
___ Yes ___ No ___ Unsure
This example of a poorly created reactionnaire has been modified from the original that appears
in "Creating Evaluation Instruments That Predict Behavior Transfer: A New Theory And
Measures In Training Evaluation" which is a Third House Inc. publication. Third House Inc.
(www.thirdhouseinc.com) has given permission for one time use in this ISPI presentation (April,
2002). Any other use is in copyright violation.
Copyright  2002 Mary L. Lanigan, Ph.D.
4 of 10
II. Create a self-efficacy section within a reactionnaire.
Example taken from the book,
"Creating Evaluation Instruments
To Predict Behavior Transfer: A
New Theory And Measures In
Training Evaluation." (Lanigan,
2001a)
For Example:
You could
even break
down this
behavior to be
more specific
regarding
which
products.
Please rate your confidence level in executing the following
behaviors.
High Confidence
Confidence
My ability to:
1. greet customers.
2. educate customers about the products.
3. close the sale.
5 4 3 2 1
Low
5 4 3 2 1
5 4 3 2 1
5 4 3 2 1
Practice: Create a self-efficacy instrument from this session's performance objectives.
 Edit corporate reactionnaires to include six assessment guidelines.
 Create a self-efficacy section within a reactionnaire.
 Use the self-efficacy section to prove content validity and internal consistency reliability.
 Predict behavior transfer using post-training self-efficacy scores.
III. Use the self-efficacy section to prove content validity and internal consistency
reliability.
A content valid instrument is an instrument that: parallels the training objectives to the evaluation
instrument items, emphasizes a proper balance between evaluation instrument items and training
objectives, and, is free from prerequisites information or skills not taught in training (Lanigan, 2001b).
For example:
Let's say you are creating a self-efficacy instrument on the training session--How To Make Coffee.
Below are the performance objectives and the time it took to deliver the content for each performance
objective.
Performance Objectives/Delivery Time
1. Open coffee can
15 minutes
3. Pour water into coffee pot
15 minutes
2. Scoop coffee into filter
30 minutes
4. Turn coffee machine on
15 minutes
EX: A content valid self-efficacy instrument for the "Making Coffee" training.
My confidence in …
1. opening a coffee can.
2. scooping coffee into a filter.
3. pouring water into the coffee pot.
4. turning the coffee machine on.
Copyright  2002 Mary L. Lanigan, Ph.D.
54132
5 4 3 2 1 (x2)
54321
54321
Notice that the questions match the
objectives; the number of test questions
are in proportion to the amount of delivery
time; and, we are not measuring anything
we have not taught in training.
5 of 10
A Content Valid Instrument
Create a content valid, self-efficacy instrument based on the performance objectives and additional
information.
Training Session on: "How To Lay Sod"
Training Objectives
Delivery Times
1.
Fertilize soil
10 minutes
2.
Place sod on soil
30 minutes
3.
Roll sod with roller
20 minutes
4.
Water sod for one hour
10 minutes
Internal consistency reliability is obtained by correlating each exam item to the total score. Internal
consistency is illustrated by Alpha. A high alpha (for example .80 or higher) gives us confidence that the
self-efficacy items are working together to generate reliable results. Measuring internal consistency
requires you to take the data obtained from the self-efficacy instrument and load it into a statistical software
package.
Step 1: Take data from self-efficacy instrument.
Step 2: Enter data into software program.
Have software multiply item 2 results by 2.
Making Coffee Self-Efficacy Instrument
My confidence in …
1. opening a coffee can is
2. scooping coffee into a filter is
3. pouring water into the coffee pot is
4. pressing the button to turn coffee
machine on is
5 4 3 2 1
5 4 3 2 1
5 4 3 2 1
5 4 3 2 1
Step 3: After entering all the data, run a reliability analysis.
Step 4: Determine the alpha level.
Reliability Coefficients 4 items
Alpha .8934
IV. Predict behavior transfer using post-training self-efficacy scores.
There are studies to support the predictive validity of using self-efficacy scores to predict behavior transfer,
and in using self-efficacy instruments as a substitute for knowledge and skills tests.
For Example:
Lanigan, M., Abbott, R. Barrett, T. and Smith, P. (in review). Making training evaluations easier by
examining the relationship between self-efficacy to knowledge, to skills and to simultaneous
pre/post instruments. Performance Improvement Quarterly.
Machin, M.A., & Fogarty, G.J. (1997). The effects of self-efficacy, motivation to transfer, and situational
constraints on transfer intentions and transfer of training. Performance Improvement Quarterly,
10(2), 98-115.
Pajares, F. (1996). Self-efficacy beliefs in academic settings. Review of Educational Research, 66(4), pp.
543-578.
Copyright  2002 Mary L. Lanigan, Ph.D.
6 of 10
Work Referenced Within This Document
Lanigan, M. (2001a). Creating evaluation instruments to predict behavior transfer:
A new theory and measures in training evaluation. Tinley Park, IL: Third House
Inc.
Recipient of a 2002 ISPI Award of Excellence for
Outstanding Instructional Communication.
Lanigan, M. (2001b). Evaluation course on-line: Reliability and validity. (Module 3).
Third House Inc. (O-line).
URL: www.thirdhouseinc.com.
V. Final questions
Copyright  2002 Mary L. Lanigan, Ph.D.
7 of 10
Appendix
Reactionnaire Example
This example of a reactionnaire has been modified from the original that appears in "Creating Evaluation Instruments
That Predict Behavior Transfer: A New Theory And Measures In Training Evaluation" which is a Third House Inc.
publication. Third House Inc. (www.thirdhouseinc.com) has given permission for one time use in this ISPI presentation
(April, 2002). Any other use is in copyright violation.
Training Evaluation: Reactionnaire
Your input regarding the questions below is most appreciated. This survey is anonymous, so
please answer honestly. Your input will be used to enhance future classes.
Section I: Self-Assessment
For each item below, please circle the response that best reflects your confidence in carrying out each skill
upon entering and exiting this class.
High
5 4 3 2 1 Low
Please use the following scale:
Confidence
Confidence
Entering this class
Exiting this class
My confidence in being able to...
5 4 3 2 1
(1) define the four levels of the traditional
training evaluation model.
5 4 3 2 1
5 4 3 2 1
(2) name the three variables that influence
behavioral intentions.
5 4 3 2 1
5 4 3 2 1
(3) name the one variable that directly
influences actual behavior.
5 4 3 2 1
5 4 3 2 1
(4) name the five tiers of the new training
evaluation model.
5 4 3 2 1
5 4 3 2 1
(5) name the evaluation instrument that is the
best predictor of actual behavior.
5 4 3 2 1
5 4 3 2 1
(6) create a self-efficacy instrument.
5 4 3 2 1
Section II: Assessment Regarding the Training Class
For each statement below, please circle the response that best represents your belief about the
training session.
Please use the following scale.
3 = Perfect 2 = Too much 1 = Too little
Facilitator Effectiveness
How would you assess the facilitator’s...
1.
projection of voice?
3
2
1
2.
rate of voice?
3
2
1
3.
energy level?
3
2
1
4.
clarity in answering questions?
3
2
1
For any item receiving a 2 or 1, please explain why you selected that response.
Copyright  2002 Mary L. Lanigan, Ph.D.
8 of 10
Adequacy of Facilities
How would you rate the training room in terms of...
5.
comfort of the chairs
(3 = comfortable, 2 = too hard, 1= too soft)
6.
7.
8.
3
2
1
temperature
(3 = comfortable, 2 = too hot, 1 = too cold)
3
2
1
ability to hear people
( 3 = perfect volume, 2 = too loud, 1 = too soft)
3
2
1
cleanliness of the restrooms
(3 = clean, 2 = too sanitary, 1 = too dirty)
3
2
1
For any item receiving a 2 or 1, please explain why you selected that response.
For each statement below, please circle the response that best represents your belief about the training
session.
Please use the following scale.
Excellent
3 2 1
Poor
Training Manual
How would you rate the training manual in terms of...
The material in Chapter 1:
9.
clarity
3
2
1
10.
visual appeal
3
2
1
11.
usefulness as a job aid when you return to the field
3
2
1
The material in Chapter 2:
12.
clarity
3
2
1
13.
visual appeal
3
2
1
14.
usefulness as a job aid when you return to the field
3
2
1
The material in Chapter 3:
15.
clarity
3
2
1
16.
visual appeal
3
2
1
17.
usefulness as a job aid when you return to the field
3
2
1
For any item receiving a 2 or 1, please explain why you selected that response.
Thank you for your participation.
Copyright  2002 Mary L. Lanigan, Ph.D.
9 of 10
Which self-efficacy instrument(s) is/are content valid based on the exercise information.
Create a content valid, self-efficacy instrument based on the performance objectives and additional
information.
Training Session on: "How To Lay Sod"
Training Objectives
Delivery Times
1.
Fertilize soil
10 minutes
2.
Place sod on soil
30 minutes
3.
Roll sod with roller
20 minutes
4.
Water sod for one hour
10 minutes
The scale for the three instruments below is:
High
5 4 3 2 1 Low
Confidence
Confidence
Self-Efficacy Instrument #1
My confidence in…
1.
fertilizing soil.
2.
placing sod on soil.
3.
rolling sod with a roller.
4.
watering sod for one hour.
54321
54321
54321
54321
Self-Efficacy Instrument #2
My confidence in…
1.
fertilizing soil.
2.
placing sod on soil.
3.
rolling sod with a roller.
4.
watering sod for one hour.
54321
54321
54321
54321
(multiply answer by 3)
(multiply answer by 2)
54321
54321
54321
54321
(multiply answer by 3)
(multiply answer by 2)
Self-Efficacy Instrument #3
My confidence in…
1.
growing sod.
2.
placing sod on soil.
3.
rolling sod with a roller.
4.
watering sod for one hour.
Copyright  2002 Mary L. Lanigan, Ph.D.
10 of 10