Multiple Choice Tests: Understanding Your Report
Once your multiple choice test has been scanned, you will receive a report summarizing student
responses and a question analysis. The question analysis refers to a varied group of statistics that
are computed for each question on a multiple choice test. These statistics can help you identify
the effectiveness of each test question. Question analysis can also help you to improve your tests
by helping to identifying ambiguous questions, and alternatives/distracters that could be more
effective.
What statistical analyses are included?
For analyses are conducted for each question on your test: proportion of responses, question
difficulty, a distractor evaluation and question discrimination.
Proportion of Responses
The proportion of responses reflects how all students
answered a given question. The value of each response
ranges between 0 and 100. When the proportion of
students selecting an answer is less than 5 per cent of
total student responses, these questions should be
reviewed to enhance their effectiveness. These will be
highlighted in yellow on your report.
Question Difficulty
Question difficulty is determined by examining the proportion of students
selecting the correct answer. Easy questions will be answered correctly by
most students and therefore have a question difficulty value that is close to
1. Very difficult questions will have a question difficulty value that is close
to 0. Questions that are effective at distinguishing between high and low
performing students will have a question difficulty near the mid point of
this range (0.5.) Include questions with a range of difficulty on your test.
Distractor Evaluation
Incorrect options or alternative answers are known as distractors.
Distractor evaluation is useful in reviewing the effectiveness of test
questions. All of the incorrect options, or distractors, should
actually be distracting. Each distractor should not be selected more
frequently than the correct answer. On your report, each distractor
should have a score less than 0, while the correct response should
have a score greater than 0. Distractors that are selected more
frequently than the correct answer, or not selected at all should be
revised to make questions more effective.
Question Discrimination
Question discrimination is the relationship between how well students did
on a question and their overall test score, and is also known as a pointbiserial correlation. Question discrimination tells you how well each
question differentiates among students in terms of their ability and
preparation. This value ranges from -1 to 1.
Effective questions have a discrimination greater than 0.20. A highly
discriminating question is one that the top 25 per cent of students scored
correctly, and the lower 25 per cent of students scored incorrectly. They
have a value closer to 1. When a question is overly difficult and/or nearly
all students answered incorrectly, the discrimination will be near 0.
Questions with a low or negative discrimination should be reviewed
carefully for confusing language or inaccuracies on the answer key, as well as how the related
topic is explored in the course. Values of 0.20 or below will be highlighted on your report.
Contact the Centre for Academic and Instructional Development, [email protected], for assistance
interpreting your multiple choice report or increasing the effectiveness of your multiple choice
tests.