Scoring and making Likert Style
questions:
This next section demonstrates how you can digitise a psychometric test on
Qualtrics. In the example we are using an adapted version of the STAIC –
“How I Feel Questionnaire,” (Speilberger & Edwards 1973) to measure the
effect different pictures have on anxiety levels.
1. To begin, make a new block and name it “How I feel questionnaire”.
2. Create a new item, and change the item style to Matrix Table.
3. Where it says “Click to write the question text,” type the questionnaire
instructions:
This questionnaire is interested in how you are feeling right now. Please select
the option which best describes how you feel. There are no right or wrong
answers. Don't spend too much time on any one statement. Remember, select
the option which best describes how you feel right now, at this very moment.
4. We now want to create our scale points. To edit the template, click
anywhere that states “click to write” and you can change that section.
Along the top of the table, edit the scale points to:
Agree
Neither Agree nor Disagree
5. Down the left hand column of the matrix
table, we want to create 20 first person
statements such as “I feel calm”. Currently,
our matrix table only has 3 spaces where we
can put statements. To change this, look at
the options that are listed on the right hand
side of Qualtrics. You should see a
“Statement” section with plus and minus
buttons. To increase the amount of
statements on the matrix table, press the plus
button until the number in this box is 20.
Equally, you can just type 20 in the box.
Disagree
6. Now we are ready to fill in the 20 statements. To edit a section, click
the text to start typing. The end result is to have a completed left
hand column containing these statements:
I feel calm
I feel frightened
I feel upset
I feel happy
I feel pleasant
I feel sure
I feel nervous
I feel good
I feel jittery
I feel troubled
I feel restless
I feel bothered
I feel scared
I feel nice
I feel relaxed
I feel terrified
I feel worried
I feel mixed-up
I feel satisfied
I feel cheerful
Scoring
Qualtrics can help with scoring responses in some ways, but not
others. Generally, when you make a psychometric test, you convert
responses such as Agree (1), Neither Agree nor Disagree (2), and
Disagree (3), into numbers for analysis purposes.
Why do we assign each response with a number?
When analysing the overall score of a psychometric test, you normally either
total up the individual responses, or calculate an average score to allow for
comparisons between people. In the current example, we are going to add up
the scores across all the questions in the STAIC “How I Feel Questionnaire”.
1. You can edit how your matrix table is coded by browsing the
advanced question options. To do this, click the “Cog” looking item
on the left hand side of your Matrix table.
2. Select “Recode Values…”
3. A new window will appear containing a rudimentary version of your
matrix table. To see how your responses are currently coded, tick
the “Recode Values” box in the yellow rectangle. This should display
numbers above your scale points. In our example, we want higher
scores to be associated with higher amounts of anxiety.
Therefore in the current example we would like Agree to be coded
as 3, Neither agree nor disagree to be coded as 2, and Disagree to
be coded as 1. You can change this by clicking in the smaller yellow
boxes above the scale points.
For every statement in the matrix table, the responses will be coded in the
same way. So if a participant Agreed with the statement “I feel calm,” and
Agreed with the statement “I feel nervous,” Qualtrics would record both of
these responses as 3. Therefore it is important to remember that the number
displayed above each scale point is applied to all of the corresponding
responses in the matrix table.
Important: Unfortunately, when using this method of coding, it is not possible
to assign different scores to different statements. For those who are familiar
with psychometric tests, you may have already figured this could be
problematic in some cases. This is because a lot of psychometric tests have
reverse questions. These are questions which are worded oppositely to
ensure that the participant is paying attention to the psychometric test. For
example in the current survey we have some statements which directly
measure anxiety: “I feel terrified,” and additionally, we also have statements
which measure the opposite: “I feel relaxed”. In these cases, Agree would
indicate a low amount of anxiety, so we wouldn’t want Agree to be coded as
3, we would want it coded as 1. Therefore, we need to find another method
that allows this.
Advanced options – Scoring
One solution to this problem is to use the “Advanced Options – Scoring”
page. On this page, you can essentially assign numbers to responses that
will be used to calculate a total end score. Note, the data that results from this
page is just a singular number: the sum of all the scale responses.
You will still get the individual responses for the matrix table, however, it will
be in the format discussed above, matching those in the “Recode Values”
window.
So what is the advantage of using this “Advanced options – Scoring” method.
First of all, when analysing psychometric tests, you often make a sum of the
responses. You can then use this number to perform a correlation with
another measure, or alternatively, you can divide it by the number of
questions in your scale to make an average. So essentially Qualtrics can
save you a job of adding up the responses yourself. Secondly, as you can
code each statement individually, the overall sum can take into account
reverse questions. Therefore this method can accurately calculate a test
score no matter how complex the scoring needs to be.
1. To begin, click “Advanced options”, then “Scoring….”
2. On the right hand side click “Scoring Options”
3. You will now see a window where you can manage
your scoring categories – A category is a group of questions i.e. a
singular psychometric test, that you would like adding together to make
a score. You would in practice, make a separate category per
psychometric test. To make a new category press New Category….
Click the box and rename the category to “STAIC”. Then click save!
4. Back on the scoring page, make sure to change the active category to
STAIC. This ensures that any editing that is done on this page is in
relation to our new category.
5. Scroll down to our “How I feel questionnaire”. Then you can start
clicking on the “-“ to add scores. To change a score from 1, simply
highlight and retype what number you would like instead.
6. Input all your scores. Remember, we want higher levels of anxiety to be
associated with higher numbers. We want lower levels of anxiety to be
associated with lower numbers.
7. You can see how this method allows you to reverse score. For
questions which directly measure anxiety i.e. “I feel upset”, you can
make agree =3, Neither Agree nor Disagree = 2 and Disagree =1. For
questions which are opposite to this and measure a state of calmness,
you can make agree =1, Neither Agree nor Disagree = 2 and Disagree
=3.
8. These inputted scores will automatically save for your scoring category
STAIC. You can see how numbers assigned to categories are
independent to one another by flicking between the categories “Score”
and “STAIC”.
The “Score” category should be blank, and the “STAIC” category should have
your inputted responses.
9. To test if you’re scoring has worked, click “Scoring Options” then click
the drop down menu next to “Show Scoring Summary for Category”
and select STAIC.
10.
Then select the
text box. Click Save.
11.
Now Press :
12.
Complete the STAIC questionnaire yourself. When you go to the
next page you should see your score:
If everything works fine then hurrah! You can decide to leave this page in the
survey if you want to show feedback to your participants. Alternatively, you
can remove this by going back into “Scoring Options” and un-ticking:
Note: When downloading your results the score will appear in its own
column: Essentially search for whatever you named your category and look
for sum. i.e. CATEGORYNAME-sum.
13. To get out of “Scoring” and back to your normal
“Edit Survey” tab, click: