SUCCESSFUL RESEARCHERS SPEND TIME
CONSTRUCTING VALID INDICATORS


As the most commonly used indicator is a survey/questionnaire,
the following power point provides detailed instructions for
construction of valid indicators using this data collection
technique.
Why is it important to construct valid indicators?
 Garbage in – Garbage OUT!
 Your indicator may appear professional, but if it is not
valid, you have garbage.
 You may have a large sample size, but if your indicator is
not valid, you have garbage.
 You may use “high powered” statistics, but if
your indicator is not valid, you have garbage.
 You may have a great program, but if your indicators are
not valid, then your evaluation of the program is
garbage.
 You may have a well-written journal article, but if you
indicator is not valid, you have garbage.
STEPS TO CONSTRUCTING A SURVEY
INSTRUMENT


Step 1 – Make Decisions about TYPES of Indicators
 Level of measurement
 Single vs. multiple indicators
 Close vs. open ended indicators
Step 2 – Consider Content of Questions
 Make it easy for respondents
 Make it easy for those who input data
 Construct indicators that yield reliable
(consistent) and valid (accurate) results

Step 3 – Consider Format of Survey/Questionnaire

Step 4 – Know How to Evaluate Surveys

Step 5 – Construct a Codebook
STEP 1 – MAKE DECISIONS ABOUT
TYPES OF INDICATORS
•
•
Level of measurement
• Measure at highest level possible.
Determined by
1. How precise can respondents be?
• For example - Can they remember
their exact income?
2. How precise are they willing to be?
• For example - Will they tell you how
many STIs they have?
Single vs. multiple indicators
• Multiple indicators should be used to
measure different dimensions of same
concept or ask about multidimensional
concept.
• Examples
1.
2.
Scales used to measure marital
happiness, depression, college readiness
(i.e., SAT, ACT) etc.
Questions used to measure different
dimensions of social class (e.g., income,
occupation, education)
STEP 1 – MAKE DECISIONS ABOUT
TYPES OF INDICATORS
•
Close vs. open-ended questions
• Two types of closed ended questions
1.
Unordered - Ask about type of
activity
Example - Which of the following
activities do you engage in before going
to bed?
1.
Reading
2.
Eating
3.
Both of the above
2.
Ordered - Ask about degree or
strength of reaction to statement
Example - Indicate how satisfied you
are with the following:
USU Extension Services
County Extension Agent
VD DS
1 2
1 2
S VS
3 4
3 4
STEP 1 – MAKE DECISIONS ABOUT
TYPES OF INDICATORS
•
Close vs. open-ended
•
Open-ended questions
•
•
•
What are the advantages?
• Important for exploratory research . For
instance, use when you do not yet know
WHAT questions to ask.
• Provide more in-depth information about
topic.
What are the disadvantages?
• Time consuming. Coding is required
once data is collected
• Validity and reliability are dependent on
the coders and their skills.
Examples of open-ended questions commonly
used in evaluation research
• What are the strengths of this program?
• How would you improve this program?
• What things have you learned from this
presentation that you can share with
others?
STEP 2 – CONSIDER CONTENT
OF QUESTIONS

Make it easy for respondents

Ask questions at the 6th grade level – 20 words or less

Avoid the word NOT in negatively worded questions (i.e.,,
I do NOT study every night.)

Avoid Acronyms (i.e., USU, ARD)

Avoid putting blanks in the middle (i.e., I read for
pleasure_____ times per week.)

Avoid having respondent rank items (i.e., rank ten most
serious crimes from high to low.)

When using contingency questions mark them clearly
with arrows.

Match questions/statements with respondents in terms
of
1.
2.
3.
Interests – Don’t ask questions about something they
have NO interest in.
Competency - Don’t ask questions about things they
know nothing about.
Relevance - Don’t ask questions about things that aren’t
relevant to them and/or their situation.
STEP 2 – EXAMPLES OF SUGGESTIONS ON SLIDE 5

Making it easy for respondent

Write at 6th grade reading level – 20 words or less
X Based on your annual net income for the previous
decade, excluding any atypical pay periods or wages
accrued by minors living in your household, what
would be your best approximate estimate of your
socio-economic status?
X
A better alternative might be
X
What is your social class?

Avoid the word NOT in negatively worded questions
 I very seldom can’t stop worrying about my grades.
 A better alternative would be
 “ I often worry about my grades.”
 This professor is not a competent teacher.
 A better alternative would be
 This professor is an incompetent teacher

Avoid Acronyms
 Do you think CEOs who graduated from USU should
attend ARDs if their child has ADD or an STI?

Avoid putting blanks in the middle
 I spend ____hours studying for exams.
 A better alternative would be
 On average, how many hours do you spend
studying for exams during a typical week?
____________ hours
STEP 2 – EXAMPLES OF SUGGESTIONS ON SLIDE 5

Making it easy for respondent

Avoid having respondent rank items
 Rank the following leisure activities from 1 to 8
based on how much you enjoy them.
__Read books __E-mail a friend
__Ride a bike
__Eat potato chips
__Swim
__Talk on my cell phone
__Watch T.V.
__ Gossip
X A better alternative might be
Please indicate your two most favorite leisure time
activities (1) ________, (2)________

Use contingency questions marked clearly with arrows.

Match questions/statements with respondents
X Interests – Don’t ask teenagers about flat taxes or
retirement pensions.
X Competency – Don’t ask the yearly income of their
great Uncle Fred.
X Relevance – Don’t ask singles about their marital
happiness.
STEP 2 – CONSIDER CONTENT
OF QUESTIONS

Make it easy for those who code, input and /or
interpret data. You cannot input letters or boxes
into the computer. The computer only recognizes
numbers.


For answer categories make certain HIGH
numbers correspond to POSITIVE answers.


THEREFORE, have respondents circle numbers for their
answers. Avoid letters and boxes.
When doing analysis, it is much easier to interpret a
number if a high number corresponds to a positive
answer. That is the way we are socialized to think about
numbers.
Use open-ended questions only when necessary.

When possible use a series of open ended questions to
get more precise answers.
STEP 2 – EXAMPLES OF SUGGESTIONS ON SLIDE 8

Making it easy for you

Have respondent circle numbers for answer categories.
Do you grow sorghum? ____ Yes _____No
A better alternative would be
Do you grow corn? (1) Yes (2) No

Have high numbers = positive answers
What grade did you receive in this class last semester?
(1) A
(3) C
(5) F
(2) B
(4) D
A better alternative would be
What grade did you received in this class last
semester?
(1) F
(3) C
(5) A
(2) D
(4) B
Please indicate how satisfied you are with the
information you received from this presenter.
Possible answers range from (1) very satisfied to
(4) very dissatisfied.
VS
S
DS
VD
1
2
3
4
A better alternative would be
Please indicate how satisfied you are with the
information you received from this presenter.
Possible answers range from (1) very dissatisfied
to (4) very satisfied.
STEP 2 – EXAMPLES OF SUGGESTIONS ON SLIDE 8

Making it easy for you

When possible use a series of open- ended
questions to get more precise answers. This
can make it much easier to code and interpret
your findings.

What do you do for a living? ________

A better alternative might be to ask a series of
questions

Where do you work? ____________________

If you work for a company or organization, what
is the name of that company/organization?
_____________

In a typical day what types of tasks do you
complete?____________________________
STEP 2 – CONSIDER CONTENT
OF QUESTIONS

Increase Validity/Reliability
 Wording

of Question
Be as Precise/specific as possible
 When asking about frequency, ask for a
specified time
 (i.e., On average, how often, per week, if
ever, do you telephone your mother? ___).

When asking for a number, ask for as precise a
number as possible
 (i.e., How much time, if any, do you spend
watching TV during an average day? ____
hours ____ minutes).

When asking for a number, try to be exact
 (i.e., How old were you on your last
birthday?____ years)
STEP 2 – CONSIDER CONTENT
OF QUESTIONS
 Increasing Validity/Reliability
 Wording
 Avoid


Slang – “hot” “cool”
Do you think it is cool to skip class?
A better alternative might be

 Avoid
•
•


of Question
Do you think students approve of skipping
classes?
Bias or Leading Questions
Cancer is a devastating illness that kills millions of
Americans every year. How much money do you think
we should spend finding a cure for this illness?
A better alternative might be
How much money do you contribute to cancer
research? ________ dollars per year.
Women who are raped never recover. They only
survive. What penalty do you think rapists
should receive?
A better alternative might be




Indicate the extent to which you agree with the
following statement s
Current penalties for convicted rapists are just.
Penalties for convicted rapists should be more
severe.
Penalties for convicted rapists are too harsh.
STEP 2 – CONSIDER CONTENT
OF QUESTIONS

Increasing Validity/Reliability
 Wording
 For

of Question
Indices
Use statements – not questions
Do you enjoy the fireworks
at the county fair?
SD D NAND A SA
1 2 3
4 5
A better alternative would be
I enjoy the fireworks
at the county fair.
SD D NAND A SA
1 2
3 4 5
STEP 2 – CONSIDER CONTENT
OF QUESTIONS

Increasing Validity/Reliability
 Wording
 For

of Question
Indices
Avoid response set
I loved this program.
The teachers were very
friendly
I learned a great deal.
I want to do this again
SD
1
D
2
A
3
SA
4
1
1
1
2
2
2
3
3
3
4
4
4
SD
1
D
2
A
3
SA
4
1
1
1
2
2
2
3
3
3
4
4
4
A better alternative would be
I loved this program
The teachers were
unfriendly
I learned nothing new
I want to do this again
STEP 2 – CONSIDER CONTENT
OF QUESTIONS

Increasing Validity/Reliability
 Wording
 For

of Question
Indices
Avoid using NOT in a negative statement
I did NOT like the
program
The teachers were NOT
friendly
I did NOT learn anything.
SD
D
A
SA
1
2
3
4
1
1
2
2
3
3
4
4
SD
1
D
2
A
3
SA
4
1
1
2
2
3
3
4
4
A better alternative would be
I disliked this program
The teachers were
unfriendly
I learned nothing new
STEP 2 – CONSIDER CONTENT
OF QUESTIONS

Increasing Validity/Reliability
 Wording
of Questions
 For
indices
 Use multiple statements to measure
multidimensional concepts (i.e., social distance)
I would live with an Asian
SD D NADA A SA
1
2
3
4 5
I would go to parties with Asians
1
2
3
4 5
I would feel uncomfortable sharing
an office with an African American
1
2
3
4 5
1
2
3
4 5
My family would disapprove if
If I married a Hispanic
STEP 2 – CONSIDER CONTENT
OF QUESTIONS

Increasing Validity/Reliability
 Wording
 For

of Questions
Indices
Avoid double barreled/double bind statements
SD
D
A
SA
1
1
2
2
3
3
4
4
1
2
3
4
A better alternative might be
SD
D
Men are poor communicators. 1
2
Women are poor communicators 1
2
I liked the staff
.
1
2
I liked the teachers
1
2
My health is better
1
2
My health is worse
1
2
A
3
3
3
3
3
3
SA
4
4
4
4
4
4
Men and women are
poor communicators
I like the staff and the
teachers.
My health has changed
since last year.

STEP 2 – CONSIDER CONTENT
OF QUESTIONS
 Wording
of Answer Categories
 Exhaustive

Cover all possible answer
Word questions so all answers are possible.
Example
How often, IF EVER, do you attend football games? ____
times per semester.
For instance
Use the answer category Other (specify)______ whenever
it is appropriate.
Word answer categories so all possible answers are
acceptable.
For instance,
Hitler was a
1.
Very negative role model
A better alternative would be
Hitler was a
1. Very negative role model
2. Negative role model
2. Negative role model
3.
4.
Positive role model
Very positive role model
STEP 2 – CONSIDER CONTENT
OF QUESTIONS
 Wording
 Use
of Answer Categories
ALWAYS and NEVER with great caution
Please indicate the extent to which you do the
following. Possible responses are (1) Never, (2)
Seldom, (3) Sometimes, (4) Often, and (5) Always
I read the newsletter mailed
out by the extension agent.
N
1
S
2
ST O A
3 4 5
A better alternative would be to replace NEVER and ALWAYS
with ALMOST NEVER and ALMOST ALWAYS.
I read the newsletter mailed
out by the extension agent.
AN S
1
2
ST
3
O AA
4 5
STEP 2 – CONSIDER CONTENT
OF QUESTIONS
 Wording
of Answer Categories
 Mutually

exclusive
NEVER have two possible right answers



For instance How much money did you make last year?
(1) $10 ,000 to $20,000
(2) $20,000 to $40,000
(3) $40,000 to $60,000
A better alternative would be
(1) $10,000 to $19,999
(2) $20,000 to $39,999
(3) $40,000 to $59,999
STEP 2 – CONSIDER CONTENT
OF QUESTIONS
 Wording
 Avoid

offensive/biased answer categories
For instance







of Answer Categories
Use overweight
Use pro-life
Use pro-choice
Use thin
Use government
assistance
NOT obese
NOT anti-abortion
NOT pro-abortion
NOT skinny
NOT welfare
For instance
What do you do when you don’t go to class?
(Circle all that apply)
1.
Act Like a Lazy Slob
2.
Waste my time
A better alternative would be
How do you spent your time when you don’t
attend class? ________________________
STEP 2 – CONSIDER CONTENT
OF QUESTIONS
 Wording
 Be

as precise/specific as possible
Use YES/NO sparingly as answer categories



of Answer Categories
Disadvantages
Doesn’t give you much variation. Is not
precise enough.
Advantages
Sometimes it is the only plausible answer.
Should you use “Don’t Know” as an answer
category?


Disadvantages
Sometimes, it is NOT a plausible answer.
Can result in response set, as
respondents may see it as the easier,
though not most accurate answer.
Advantages
Can add important variation to possible
answers, and REMEMBER you need/
want variation when you start doing
analysis.
STEP 3 – CONSIDER FORMAT OF
SURVEY

General Guidelines
 Using Numbers

For single items – number from top to bottom
in columns.
 What grade did you receive?
1. F
4. B
2. D
5. A
3. C

For index : (1) put numbers under symbols,
(2) number from left to right (3) number from
lower to higher
SD
1
1

D
2
2
NAD
3
3
Professional Appearance




A
4
4
Spacing – even and balanced
Uncluttered appearance
No spelling or grammar errors
Can use shading etc.
SA
5
5
STEP 3 – CONSIDER FORMAT
OF SURVEY

Required Sections

Textbox

Should contain the following information

General statement about the purpose of
survey

Statement indicating the time survey will take

Statement that indicates it is anonymous or
confidential

Statement that indicates it is voluntary

Contact information for those wanting more
information

Basic sociodemographic questions

Questions measuring indicators/variables

Contingency questions

All parts should be on the same page

Indices/Scales

The entire index/scales should be on same page –
if possible. If you put it on two pages, then repeat
headings.
PLACEMENT OF SECTIONS
Section
Placement
Textbox
Top of first page of survey
Sociodemographic
First page of survey
Questions measuring indicators
Sensitive questions
Independent variable
Dependent variable
Open-ended questions
Bottom of last page of survey
First page of survey
Second page of survey
End of survey
Contingency Questions
Throughout the survey
Index (ices)
Generally middle or end
of survey
STEP 4 - CRITERIA FOR EVALUATION QUESTIONS YOU MUST ASK

Appearance




Statement at Top



Does it look professional?
Is the text “balanced”?
Did you use numbers? Are they in correct order?
Does it include relevant/complete information?
Is it in a textbox?
Questions

Format – for index or scale





Does it have a textbox at top with instructions?
Is it numbered from left to right – lower to higher?
Did you use statements – not questions?
Are statements logically related to each other?
Wording



Did you use a spelling and grammar check?
Did you avoid biased statements/words, slang,
abbreviations, double-barreled questions/statements etc.?
Answer categories must be:
-measured at highest level possible?
-exhaustive/mutually exclusive?

Did you test your hypotheses/measure your indicators?
AVOID COMMON MISTAKES






Use numbers not letters
Measure indicators/variables at the highest level possible (i.e.,
the ratio level, whenever possible)
Ask only relevant questions
 Don’t ask college students if they graduated from high
school
Use statements rather than questions in indices. You can’t
agree with a question!
Avoid bias. If you can predict what answer the researcher
“wants” then it is a biased statement or question.
Make wording across answer categories consistent
 For instance, IF you use pro-life, then use pro-choice
 For instance, IF you use African American, then use Asian
American.
EVALUATING YOUR INDICATORS
You can use the following chart or one like it to evaluate the validity of
the indicators used in your survey/questionnaire
Concept
Instrument
Face Validity (Y/N)
For each indicator, ask yourself the following question?
1.
Can you match a question or questions on your survey
with each indicator you are using to measure impact?
2.
Are you measuring attitudes or behavior? Does the
question/statement in the survey match?
2.
If you are measuring a multidimensional concept – do
you have more than one question/statement?
Then determine if your concept has face validity (i.e., it logically
appears to measure the concept).
EVALUATING YOUR INDICATORS
EXAMPLE
You can use the following chart or one like it to evaluate the validity of the
indicators used in your survey/questionnaire
Concept _________
Instrument
_________
Face Validity (Y/N)
Impact of program
on recycling behavior
How often do you recycle
your drink containers?
Y
Impact on attitudes toward
water conservation
To what extent do you agree
that Utah residents must
learn to conserve water or
face shortages.
Y
For each indicator, ask yourself the following question:
1.
Can you match a question or questions on your survey
with each indicator you are using to measure impact?
2.
Are you measuring attitudes or behavior? Does the question/statement
in the survey match?
2.
If you are measuring a multidimensional concept – do
you have more than one question/statement?
Then determine if your concept has face validity (i.e., it logically appears
to measure the concept).
EVALUATING YOUR INDICATORS
You can use the following chart to evaluate the reliability of each of
your indicators. Take each concept, and then rate the
indicator on the following characteristics. Generally, the
higher the score, the more reliable the indicator.
Indicator 1
Precise
Unbiased
Not Double barreled
Exhaustive
Exclusive
Not offensive
Reliability
(+/-)
For each of your indicators, ask yourself if you think your indicators
would yield consistent results across time. Answer the
following questions:
1.
2.
3.
4.
5.
6.
7.
Are questions precise and unambiguous?
Are there any words that may be interpreted differently by
different people? (slang)
Are there biased questions that may influence some
people’s answers?
Are any questions double barreled?
Are answers exhaustive? (Did you use “Other – please
specify” as an answer category?)
Are answers mutually exclusive?
Are there questions that may offend someone?
EVALUATING YOUR INDICATORS – AN EXAMPLE
Use the following chart to evaluate the reliability of your indicators.
Take each concept, and rate the indicator.. Generally, the higher the
score, the more reliable the indicator.
Indicator 1
Reliability
How would you rate the presentation
you just attended?
1.
Fair
2.
Good
3.
Excellent
Precise
Unbiased
Not Double barreled
Exhaustive
Exclusive
Not offensive
(+/-)
+
+
+
Only 3 of 6. This indicator needs to be reworded
For each of your indicators, ask yourself if you think your indicators would yield
consistent results across time. Answer the following questions:
1.
2.
3.
4.
5.
6.
7.
Are questions precise and unambiguous?
Are there any words that may be interpreted differently by different
people? (slang)
Are there biased questions that may influence some people’s answers?
Are any questions double barreled?
Are answers exhaustive? (Did you use “Other – please specify” as an
answer category?)
Are answers mutually exclusive?
Are there questions that may offend someone?
STEP 5 – CONSTRUCTING A CODEBOOK
General Guidelines

Identify a name for your codebook. It should start with a
letter (not a number), and should contain no slashes,
hyphens etc. It should be about 8 characters.

Put an identification number on each of your surveys. This
identification number should be the first variable in your
codebook.

Construct a name to identify each statement/question used
in your survey to measure each indicator. Do not use
slashes/dashes/spaces. This name will appear on your
statistical outputs when analyzing this question/statement.

Compose a BRIEF description for each variable/indicator.

Construct a code that corresponds to each
variable/indicator. The code should consist of numbers
representing each possible response/answer.
STEP 5 – CONSTRUCTING A CODEBOOK
FORMAT OF CODEBOOK
Q # Var#
1
Var Name
Description
___ Code_____
1
ID
Identification #
1-100
2
Sex
Sex of respondent
1=male
2=female
3
Age
Age of Respondent
18-99 years
4
Fear
“I fear my future
1=strongly disagree
has few
2=disagree
3=undecided
4=agree
5=strongly agree
5
Hope
I have great
hope for my future
same
EXAMPLES OF SURVEYS

Following is an Example of a Survey.

Please note the following:


There is no textbox at the beginning.
 The information often provided in the textbox
was provided in a cover letter.
There are a number of different indices


Some of the questions indicate that you can circle
all that apply


This ensures that possible answer categories are
exhaustive.
Some questions have an “other – specify”
category.


You don’t have to always use Strongly Disagree to
Strongly Agree.
This ensures possible answer categories are exhaustive.
Not all of the questions/statements are written
correctly
This should be
from 1. Mostly F’s
to 5. Mostly A’s.
IMPORTANT FINAL COMMENTS

ALWAYS DO
 Have
others proofread your survey and
give you suggestions
 Pre-test
your survey using respondents
similar to those in your sample
 Submit
your survey to IRB for approval
 Let
respondents know you are associated
with USU

NEVER DO
 Send
our surveys without IRB approval
FOR MORE INFORMATION

Contact Dr. Carol Albrecht
Assessment Specialist
USU Extension
 Phone
– 979-777-2421
 E-mail – [email protected]