Module
6
Assessment of Student Learning
Module 6
ITEM ANALYIS
Objectives

Discuss the importance of item analysis in test development

Compute difficulty index and discrimination index of test items

Interpret item analysis data
Introduction
The preceding two lessons dealt with the different methods of assessment that
can be used to assess varying learning targets. After writing the items, a test
cannot be considered good unless it is tried out because the ultimate judge of the
test is the user. More often than not, teachers prepare, administer, and score a
test, return the papers to their students, possibly discuss the test, and then either
file or discard the test. Ebel (1965) believes that one of the common mistakes of
teachers is that they do not check on the effectiveness of the their test because of
the following reasons: a) teachers feel that test analysis is too time consuming, b)
they are not aware of the methods of analyzing tests, and 3) they do not always
understand the importance of accurate evaluate evaluation.
Page 1 of 14
Module
6
Assessment of Student Learning
Item analysis
Remember that tests are used for instructional purposes and other important
decisions, it is therefore important to conduct item analysis. Item analysis is the
process of evaluating the quality of the items and of the tests as a whole. Item
analysis “investigates the performance of items considered individually in relation
to the remaining items on the test” (Thompson & Leviton, 1985). It is the process
of examining the pupil’s responses to each test item.
The item analysis procedure provides the following information:

The difficulty of the item

The discriminating power of the item

The effectiveness of the distracters
The key aim of the item analysis is to improve the whole test and
eventually increase the reliability and reliability of the test. The tools to test the
validity and reliability of the test include item difficulty, item discrimination, and
effectiveness of distracters. There are many methods that can be used for item
analysis. One method is the U – L Index Method (Stocklein, 1957). The steps
are:
1.
Score the papers and rank them from highest to lowest according to
total score.
2. Separate the top 27% and the bottom 27% of the papers.
Page 2 of 14
Module
6
Assessment of Student Learning
3. Tally the responses made to each test item by each individual in the
upper 27% group.
4. Tally the responses made to each test item by each individual in the
lower 27% group.
5. Compute the percentage of the upper group that got the item right and
call it U.
6. Compute the percentage of the lower group that got the item right and
call it L.
7. Average U and L percentage and the result is the difficulty index of
the item.
8. Subtract the L from the U percentage and the result is the
discrimination index of the item
Item difficulty/difficulty index
Difficulty index, denoted by p, is simply the percentage of the students taking the
test who answered the item correctly. It can be interpreted as how easy or how difficult
an item is. The larger the percentage getting an item right, the easier the item. The
higher the difficulty index, the easier the item is understood to be. For example, an item
answered correctly by 85% of the examinees will have an item difficulty of .85, whereas
an item answered by 50% of the examinees would have a lower item difficulty of .5. it is
usually best when difficulty index is around .5, for it provides maximum differentiation.
The best difficulty index, however, is halfway between the lowest and highest expected
score. When all of the items are extremely difficult, the great majority of the test scores
Page 3 of 14
Module
6
Assessment of Student Learning
will be very low. When all items are extremely easy, most test scores will be extremely
high. We do not want items that are too easy or too difficult.
Item discrimination/discrimination index
A good item discriminates between those who do well on the test and those who
do poorly (Susan Matlock-hetzel, 1997).
The discriminative power of a test item,
denoted by D refers to the degree of the ability of an item to distinguish those who know
from those who do not know. It can be measured by comparing the number of people
with high test scores who answered that item correctly with the number of people with
low scores who answered the same item correctly. The top and bottom 27% are used for
analysis because 27% has shown that this value will maximize differences in the normal
distribution while providing enough cases for analysis (Wiersma and Jurs (1990). Other
methods use 30% while others use 50% (no middle group).
The higher discrimination index, the better the item because such a value
indicates that the item discriminates in favor of the upper group, which should get more
items correct. If more students in the lower group get an item correct than in the upper
group, the item will have a negative D value and is probably flawed. Similarly, if there
are equal number of students from the upper group and lower group, the item cannot
discriminate because D value is 0.
Page 4 of 14
Module
6
Assessment of Student Learning
Computing for the difficulty index and discrimination index of an item is a lot
easier than interpreting it. Remember that the purpose of the analysis is to determine
which are good items; that is there are items to be retained or accepted, rejected or
discarded, and revised. The table below will serve as guide in interpreting the results of
the item analysis.
Table for interpreting difficulty index (p)
Range of Difficulty Index (p)
Interpretation
0 - .20
Very difficult
0.21 – 0.40
Difficult
0.41 – 0.60
Moderately difficult
0.61 – 0.80
Easy
0.81 – 1.00
Very easy
Table for interpreting discrimination index (D)
Range of Discrimination Index (D)
Interpretation
-1.0
Questionable
- -.06
-0.59 - -0.20
Not discriminating
-0.21 - 0.20
Moderately discriminating
0.21
Discriminating
- 0.60
0.61 -
1.00
Very discriminating
In interpreting the results, we have to consider not only whether how easy or how
difficult the item is, but also its ability to discriminate students who know and those who
do not know the answer. In other words, both the p values and D values are taken into
Page 5 of 14
Module
6
Assessment of Student Learning
consideration. The decision rule is to retain or accept the items that are not so easy or too
difficult, and at the same time can discriminate bright from poor students.
After interpreting the difficulty and discrimination indices, the table below will
help us what to do with the test item
Difficulty Index
Difficult
Discrimination Index
Suggested Action
Not discriminating
Discard/Reject
Moderately discriminating
May need revision
Discriminating/Very
Accept/Retain
discriminating
Moderately difficult
Not discriminating
Needs revision
Moderately discriminating
May need revision
Discriminating/Very
Accept/Retain
discriminating
Easy
Not discriminating
Discard/Reject
Moderately discriminating
Needs Revision
Discriminating/Very
Needs Revision
discriminating
Note that whether an item is easy, moderately difficult or difficult, there are
always three categories of discrimination indices that can be obtained, that also leads to
three different actions. For example, an easy item could either be not discriminating,
moderately discriminating or discriminating, and then it could be rejected, revised or
retained.
Page 6 of 14
Module
6
Assessment of Student Learning
However, care and caution musts be followed in using the table in interpreting the
results of an item analysis. Judgment of the test constructor sis very important. For
example, what will be done with an item that is easy and not discriminating. Using the
table, we should reject the item. But there will be an instance when that kind of item can
be revised. When? When that particular item is the only item left to test a very important
concept. So, we have no choice but to revise or improve it. On the other hand, what will
be done with an item that is moderately difficult and discriminating? Normally that item
should be retained because it has good indices. But there will be instance when that kind
of item may be discarded or rejected. That will happen if there are already enough items
to test the particular concept or skill that it assesses.
The table below shows a sample result of an item analysis, illustrating the steps of
the U-L Index method.. Study the results and focus on the interpretation of the test item.
Table showing a summary of the Item Analysis
Ite
Uppe
Lowe
m
r
r
No.
27%
27%
1
14
12
U
L
p
Interpretation
D
Interpretation
Action
.88
.75
.82
Very easy
.13
Moderately
Revise
discriminating
2
10
6
.63
.38
.51
Moderately
.25
Discriminating
Retain
.25
Discriminating
Retain
.43
Discriminating
Retain
difficult
3
11
7
.69
.44
.57
Moderately
difficult
4
9
2
.56
.13
.35
Difficult
Page 7 of 14
Module
6
Assessment of Student Learning
5
12
6
.75
.38
.57
Moderately
.37
Discriminating
Retain
difficult
6
6
14
.38
.88
.63
Easy
-.50
Questionable
Discard
7
13
4
.81
.25
.53
Moderately
.56
Discriminating
Retain
-.44
Questionable
Discard
.06
Moderately
Revise
difficult
8
3
10
.19
.63
.41
Moderately
difficult
9
13
12
.81
.75
.78
Easy
discriminating
10
8
6
.50
.38
.44
Moderately
difficult
.12
Moderately
discriminating
Number of students tested - 60
Step 1. The papers have been scored and arranged from highest to lowest
Step 2. There were 60 students who took the test, so 27% of 60 is 16. Therefore, there
are 16 students
from the upper group and another 16 students for the lower
group. The middle group (28 students) will not be included in the analysis.
Step 3. The number of students from the upper group was tallied. For example, out of 16
students, 14 students got item 1 correctly, 10 students for item 2, and so forth.
Step 4. The number of students from the lower group was tallied. For example, out of 16
students, 12 students got item 1 correctly, 6 students for item 2, and so forth.
Step 5. The percentage of the students from the upper group is obtained by dividing the
number of students who got the item right and the total number of students; that is
14/16 = .88, or 88%., 10/16 = .63 or 63 % for item 2, and so forth. The results are
placed under Column U (upper group)
Step 6. The same procedure like in step 5 was performed. For example, 12/16 = .75 0r
755 for item 1, and so forth. Results are placed under Column L (lower group)
Page 8 of 14
Revise
Module
6
Assessment of Student Learning
Step 7. The difficulty index (p) was computed by getting the average of U and L
percentages. For example in item 1 U = .88, L = .75. Therefore, p = (U+L)/2 =
(.88+.75)/2 = .82. This means that 82 % of the students got item 1 correctly.
Step 8. The discrimination index (D) is obtained by subtracting L from U, that is D = U –
L, or D = .88 - .75 = .13
So for item number 1, the difficulty index is .82 and the discrimination index is
.13. Using the tables presented earlier, a difficulty index of .82 is said to be very easy,
and a discrimination index of .13 is moderately discriminating. The item therefore needs
revision. The item was too easy because there were many students, even from the lower
group who got the item correctly.
Item Distractor Analysis
An item distractor analysis is performed to review items judged to be problematic
during difficulty and discrimination analyses, to evaluate the plausibility of distracters or
incorrect options, and to identify areas in which instruction needs to be revised. For
example, item number 1 in the previous item analysis needs revision. Why? Because the
item is very easy, though it can moderately discriminate. In selection-type of test, like
the multiple choice, an item may be very easy or very difficult because of the options or
alternatives.
Page 9 of 14
Module
6
Assessment of Student Learning
To perform distractor analysis, there is a need to compare the selected responses
of students in the upper and lower group. Creating a distractor analysis table will help
summarize this information. Look at the example below:
Item Number
1
9
Options
A
B
C
D
Upper
0
14**
2
0
Lower
0
12**
4
0
Upper
13**
1
1
1
Lower
12**
2
2
0
Upper
8**
8
0
0
Lower
6**
10
0
0
10
N = 16
** correct answer/key
For example, item number has p value of .82 and D value is .13. The item is easy
but discriminating, which means that the item needs revision. If we will revise the item,
we might look as well at the distracters. Not one of the respondents choose A and D.
This may have two explanations. The distracters are impossible or illogical to them.
You may have included distracters that are obviously wrong. However, if you really
think that A and D represent logical alternatives to the correct answer, then it is more
probable that B reflects common misconception held by large portion of the class. This
Page 10 of 14
Module
6
Assessment of Student Learning
requires revising your initial instruction and includes information on how to avoid the
problem. The same thing applies to item number 9. For item number 10, the item needs
revision as well. Though the item can moderately because there were more students from
the upper group than the lower group who got the correct answer, the same number of
students from the upper group chose a distracter as the correct answer (option B).
Similarly, looking at the lower group, it seems that option B is the correct answer for item
10.
In summary, item analysis provides relevant information about the individual
items of the test, and the test as a whole. Always remember that you can maximize the
ability of the test to help you assess student’s learning by making sure that you construct
high quality test items.
Page 11 of 14
Module
6
Assessment of Student Learning
Activity 1
A. Complete the table below.
Ite
Uppe
Lowe
m
r
r
No.
27%
27%
1
20
18
2
19
12
3
17
11
4
10
20
5
21
11
6
9
2
7
24
14
8
18
13
9
9
19
10
22
15
11
26
24
12
25
13
13
6
3
14
23
12
15
11
19
U
L
p
Interpretation
D
Interpretation
Number of pupils tested = 10
Page 12 of 14
Action
Module
6
Assessment of Student Learning
Activity 2
A. Answer the following questions.
1. When do we obtain a difficulty index of 0.0?; 1.0?
2. When do we get a negative discrimination index?; positive discrimination index?;
zero discrimination index?
3. If the item has a perfect discrimination index, what is the difficulty index of that
item?
4. In as far as item analysis is involved, what is a “good item”?
B. Analysis of Distractors
Below shows the students answers in 5 items. Study the information and
in each item, indicate (under remarks row) whether each option is a good, poor or
fair option. Figure with asterisk indicate the correct response.
Item 1
A
B
C
D
Upper
10*
4
1
1
Lower
7
0
4
5
Item 2
A
B
C
D
Upper
5*
5
5
1
Lower
16
0
0
0
Remarks
Remarks
Page 13 of 14
Module
6
Assessment of Student Learning
Item 3
A
B
C
D
Upper
2
3
4
7*
Lower
8
2
5
1
Item 4
A
B
C
D
Upper
2
8*
1
5
Lower
5
5
1
5
Item 5
A
B
C
D
Upper
16*
0
0
0
Lower
16
0
0
0
Remarks
Remarks
Remarks
Activity 3
Share insights gained from the lesson (2-3 short paragraphs)
Page 14 of 14