Sumber :
http://www.usq.edu.au/library/help/postgrad/resmeth.htm
http://www.tardis.ed.ac.uk/~kate/qmcweb/q2.htm
Dealing with Problem Questions
Hypothetical questions
A hypothetical question is one in which you are asking respondents to indicate what they
think they would do under particular imaginary circumstances. These can't always be
avoided in some attitudinal research, but they are difficult to administer and often give
rise to unreliable answers
Presuming / leading questions
These are often included in poor questionnaires because the researcher feels strongly
about a topic and assumes that everyone will be of the same opinion.
Questions which rely on memory
Problems which tax the respondent's memory too much are likely to lead to non-response
or inaccurate replies. For example "What did you have for lunch each day last week?"
Questions requiring prior knowledge
For example, "What is your National Insurance number?"
Sensitive questions

Personal details / health / age

Income
If you have to ask sensitive questions, the problem can be alleviated somewhat by the use
of SHOW CARDS. Put all of the possible responses on a card, preferably mixed up, and
ask the respondent to indicate which number relates to their own circumstances. For
example,
Can you tell me the number on this card which corresponds to you income group?
SHOW CARD WITH……
1.
2.
3.
4.
5.
6.
7.
£7,000 - 12,000
Over £60,000
£18,000 - £30,000
Under £7,000
£40,000 - £60,000
£12,000 - £18,000
£30,000 - £40,000
Mutually exclusive responses
In the show card above, you will note that somebody earning exactly £30,000 would
perhaps wonder whether to give answer 3 or answer 7 on the show card. In practice,
people are usually able to give their income as an approximation. You should, however,
always watch out for questions where the multiple choice answers are not mutually
exclusive and where a respondent will be uncertain about which category he/she falls
under. It seems to be a particular problem with age brackets, and you can often see
examples of mistakes here in even professionally produced surveys.
Long questions
If your questions are too long and detailed, the respondent may get lost and the responses
will relate only to the beginning or the end of the question. Where definitions and
qualifications are necessary, use show cards.

Analysis of Survey Data
Influenced by:
1. The number of variables being examined



Univariate analysis = one variable (e.g. gender, age) Methods: charts (e.g. bar
chart, pie chart) or frequency table
Bivariate = 2 variables (e.g. gender + purchase of computers) Methods:
crosstabulations; scatterplots; regression; comparison of means (but we will only
be covering the first two in this module)
Multivariate = 3 or more variables (e.g. income influenced by both education
level and gender) - Note: you'll be relieved to hear that you won't be covering any
of these as part of this module)
2. Levels of measurement




Nominal
Ordinal
Interval
Ratio
3. Descriptive or inferential?


Descriptive statistics: summarise patterns in responses (e.g. average age of
respondents or number of respondents who buy a product)
Inferential statistics: provide an idea about whether the patterns described in the
sample are likely to apply in the population from which the sample is drawn
("tests of significance")
Frequency tables
Should have separate columns for:


(a) number of people who gave each answer (N)
(b) percentage column
If (a) is excluded from the table, it is essential to include the total number of cases on
whom percentages are calculated.
Example
Analysis of degree of agreement/disagreement with statement "Taking part in
videoconferenced classes is a really exciting experience"
Answer
N
%
Strongly agree
Agree
Neutral
Disagree
Strongly Disagree
Total
254
201
119
97
75
746
34
27
16
13
10
100
Crosstabulation





Crosstabulation = tabular representation of relationship between two (or can be
more than two) variables
Also called CONTINGENCY TABLE
Table divided into cells with each cell representing the coincidence of a specific
value from each variable
The total for each row and each column is given at the end of the row and column.
These totals are called the ROW MARGINALS and COLUMN MARGINALS.
It is normal (but not absolutely essential) for the dependent variable to appear in
rows. This is because we may be crosstabulating one dependent variable by a
series of possible explanatory variables.
Hypotheses & Variables
A hypothesis can be described as "a tentative answer to a research question" or a
"provisional prediction". The plural is hypotheses.
Hypotheses should be:




Stated clearly, using appropriate terminology
Testable
Statement of relationships between variables
Limited in scope
Examples of hypotheses:




Health education programmes influence the number of people who smoke
Newspapers affect people's voting patterns
Attendance at lectures influences exam marks
Diet influences intelligence
In the above examples, "something" (e.g. diet, lecture attendance) affects "something
else" (e.g. intelligence, exam marks). These are variables. A variable is anything which is
free to vary, and in order to describe them quantitatively, they have to be expressed in
appropriate units (e.g. IQ scores, exam percentages)
The pairs of variables in the above examples have separate names. The variable we
manipulate is called the independent variable (IV). The variable we are hypothesising
will alter as a result of our manipulations is called the dependent variable (DV). The
dependent variable alters as a consequence of the value of the independent variable - its
value is dependent on this. The value of the independent variable is free to vary according
to the whims of the experimenters.
Independent variable
 Health education programmes
 Newspaper
 Attendance at lectures
 Diet
Dependent variable
 Number of people who smoke
 Voting patterns
 Exam marks
 Intelligence
NB Many variables can be either dependent or independent, within the context of a
particular study. For example, it could be argued that "Intelligence influences diet" or
"Exam marks influence attendance at lectures".
Directional hypotheses
In the examples above, words like "influence" or "affects" are used without indicating
direction. When a hypothesis states a predicted outcome (using words such as reduce,
increase, lower, raise - it is called a directional or one-tailed hypothesis. Vaguer types
of hypotheses (such as the ones given earlier) are known as non-directional or twotailed hypotheses.
Sumber :
http://www.usq.edu.au/library/help/postgrad/resmeth.htm
http://www.tardis.ed.ac.uk/~kate/qmcweb/q2.htm
Question Order

Your respondents may refuse to co-operate if your survey begins with awkward or
embarrassing questions.


People are more likely to give honest replies to personal questions if some rapport
has been developed with the interviewer.
For the above reasons, it is generally best to keep all questions dealing with
demographic information (such as age) at the end of the questionnaire.
The Layout of the Questionnaire



Print clearly (if computer printout is giving faint printouts, ask the IT staff
whether a new print cartridge can be installed)
Allow adequate space between questions so that you can write down any
comments made (but don't waste too much paper!!)
Write the questions themselves in lower case (i.e. like this writing) ,
INSTRUCTIONS IN UPPER CASE (i.e. capital letters)
Dealing with Problem Questions
Hypothetical questions
A hypothetical question is one in which you are asking respondents to indicate what they
think they would do under particular imaginary circumstances. These can't always be
avoided in some attitudinal research, but they are difficult to administer and often give
rise to unreliable answers
Presuming / leading questions
These are often included in poor questionnaires because the researcher feels strongly
about a topic and assumes that everyone will be of the same opinion.
Questions which rely on memory
Problems which tax the respondent's memory too much are likely to lead to non-response
or inaccurate replies. For example "What did you have for lunch each day last week?"
Questions requiring prior knowledge
For example, "What is your National Insurance number?"
Sensitive questions


Personal details / health / age
Income
If you have to ask sensitive questions, the problem can be alleviated somewhat by the use
of SHOW CARDS. Put all of the possible responses on a card, preferably mixed up, and
ask the respondent to indicate which number relates to their own circumstances. For
example,
Can you tell me the number on this card which corresponds to you income group?
SHOW CARD WITH……
8. £7,000 - 12,000
9. Over £60,000
10. £18,000 - £30,000
11. Under £7,000
12. £40,000 - £60,000
13. £12,000 - £18,000
14. £30,000 - £40,000
Mutually exclusive responses
In the show card above, you will note that somebody earning exactly £30,000 would
perhaps wonder whether to give answer 3 or answer 7 on the show card. In practice,
people are usually able to give their income as an approximation. You should, however,
always watch out for questions where the multiple choice answers are not mutually
exclusive and where a respondent will be uncertain about which category he/she falls
under. It seems to be a particular problem with age brackets, and you can often see
examples of mistakes here in even professionally produced surveys.
Long questions
If your questions are too long and detailed, the respondent may get lost and the responses
will relate only to the beginning or the end of the question. Where definitions and
qualifications are necessary, use show cards.
Analysis of Survey Data
Influenced by:
1. The number of variables being examined



Univariate analysis = one variable (e.g. gender, age) Methods: charts (e.g. bar
chart, pie chart) or frequency table
Bivariate = 2 variables (e.g. gender + purchase of computers) Methods:
crosstabulations; scatterplots; regression; comparison of means (but we will only
be covering the first two in this module)
Multivariate = 3 or more variables (e.g. income influenced by both education
level and gender) - Note: you'll be relieved to hear that you won't be covering any
of these as part of this module)
2. Levels of measurement




Nominal
Ordinal
Interval
Ratio
3. Descriptive or inferential?


Descriptive statistics: summarise patterns in responses (e.g. average age of
respondents or number of respondents who buy a product)
Inferential statistics: provide an idea about whether the patterns described in the
sample are likely to apply in the population from which the sample is drawn
("tests of significance")
Frequency tables
Should have separate columns for:


(a) number of people who gave each answer (N)
(b) percentage column
If (a) is excluded from the table, it is essential to include the total number of cases on
whom percentages are calculated.
Example
Analysis of degree of agreement/disagreement with statement "Taking part in
videoconferenced classes is a really exciting experience"
Answer
Strongly agree
Agree
Neutral
Disagree
Strongly Disagree
Total
N
254
201
119
97
75
746
%
34
27
16
13
10
100
Crosstabulation





Crosstabulation = tabular representation of relationship between two (or can be
more than two) variables
Also called CONTINGENCY TABLE
Table divided into cells with each cell representing the coincidence of a specific
value from each variable
The total for each row and each column is given at the end of the row and column.
These totals are called the ROW MARGINALS and COLUMN MARGINALS.
It is normal (but not absolutely essential) for the dependent variable to appear in
rows. This is because we may be crosstabulating one dependent variable by a
series of possible explanatory variables.
Hypotheses & Variables
A hypothesis can be described as "a tentative answer to a research question" or a
"provisional prediction". The plural is hypotheses.
Hypotheses should be:




Stated clearly, using appropriate terminology
Testable
Statement of relationships between variables
Limited in scope
Examples of hypotheses:


Health education programmes influence the number of people who smoke
Newspapers affect people's voting patterns


Attendance at lectures influences exam marks
Diet influences intelligence
In the above examples, "something" (e.g. diet, lecture attendance) affects "something
else" (e.g. intelligence, exam marks). These are variables. A variable is anything which is
free to vary, and in order to describe them quantitatively, they have to be expressed in
appropriate units (e.g. IQ scores, exam percentages)
The pairs of variables in the above examples have separate names. The variable we
manipulate is called the independent variable (IV). The variable we are hypothesising
will alter as a result of our manipulations is called the dependent variable (DV). The
dependent variable alters as a consequence of the value of the independent variable - its
value is dependent on this. The value of the independent variable is free to vary according
to the whims of the experimenters.
Dependent variable
 Number of people who smoke
 Voting patterns
 Exam marks
 Intelligence
Independent variable
 Health education programmes
 Newspaper
 Attendance at lectures
 Diet
NB Many variables can be either dependent or independent, within the context of a
particular study. For example, it could be argued that "Intelligence influences diet" or
"Exam marks influence attendance at lectures".
Directional hypotheses
In the examples above, words like "influence" or "affects" are used without indicating
direction. When a hypothesis states a predicted outcome (using words such as reduce,
increase, lower, raise - it is called a directional or one-tailed hypothesis. Vaguer types
of hypotheses (such as the ones given earlier) are known as non-directional or twotailed hypotheses.
Sumber : http://www.researchmethods.com/
Reliability
I. The problem with random error
A. Plays havoc with individual scores
B. Like static or unwanted noise, hides the message that the data are sending you--but
doesn't bias the results.
C. Sources of random error
1. Observers
2. Testing situations vary
3. Individuals vary (weight, mood, etc. fluctuate)
II. Assessing random error to see if it is a problem
A. The technique for assessing random error is based on one assumptions about the
nature of people: underlying traits and behavior are basically stable and consistent.
B. Test-retest reliability coefficient ranges from 0-1, with higher numbers indicating
higher reliability
1. What to do if it's over .90
2. What to do if it's under .60
III. Assessing random error due to the observer
A. Interjudge agreement
B. Inter-observer reliability
1. ranges from 0-1
2. puts a ceiling (but not a floor) on test-retest reliability
IV. What to do if there is too much random error due to the observer
A. Train and motivate raters
B. Use instruments
V. Dealing with random error due to non-observer sources
A. Reduce random error due to the testing situation by standardizing how the measure is
administered
B. Reduce random error due to the participant by
1. Interviewing participants to find out if certain questions are confusing or hard to
answer. Since participants may be guessing on those items, that guessing will lead to
random error. Consequently, deleting those items may increase reliability.
2. Using more items or larger samples of behavior so that random error will have more
chances to balance out. That is, scores on a multiple-choice test made up of 1 item will be
more influenced by random error than a 100-item test.
VI. Summary:
A. Reliability does not guarantee validity; it is only a prerequisite for validity. A reliable
measure can be consistently measuring (or being influenced by) the wrong thing, such as:
1. Another construct
2. Observer bias
B. Low reliability waters down, but does not poison, validity.
Sumber : http://www.researchmethods.com/
Reliability
I. The problem with random error
A. Plays havoc with individual scores
B. Like static or unwanted noise, hides the message that the data are sending you--but
doesn't bias the results.
C. Sources of random error
1. Observers
2. Testing situations vary
3. Individuals vary (weight, mood, etc. fluctuate)
II. Assessing random error to see if it is a problem
A. The technique for assessing random error is based on one assumptions about the
nature of people: underlying traits and behavior are basically stable and consistent.
B. Test-retest reliability coefficient ranges from 0-1, with higher numbers indicating
higher reliability
1. What to do if it's over .90
2. What to do if it's under .60
III. Assessing random error due to the observer
A. Interjudge agreement
B. Inter-observer reliability
1. ranges from 0-1
2. puts a ceiling (but not a floor) on test-retest reliability
IV. What to do if there is too much random error due to the observer
A. Train and motivate raters
B. Use instruments
V. Dealing with random error due to non-observer sources
A. Reduce random error due to the testing situation by standardizing how the measure is
administered
B. Reduce random error due to the participant by
1. Interviewing participants to find out if certain questions are confusing or hard to
answer. Since participants may be guessing on those items, that guessing will lead to
random error. Consequently, deleting those items may increase reliability.
2. Using more items or larger samples of behavior so that random error will have more
chances to balance out. That is, scores on a multiple-choice test made up of 1 item will be
more influenced by random error than a 100-item test.
VI. Summary:
A. Reliability does not guarantee validity; it is only a prerequisite for validity. A reliable
measure can be consistently measuring (or being influenced by) the wrong thing, such as:
1. Another construct
2. Observer bias
B. Low reliability waters down, but does not poison, validity.