Project Quantum ethos for giving and receiving feedback
November 2016
Summary
Devising multiple choice questions is not easy! Quantum (via the DQ system) provides a
way for teachers to offer constructive feedback to the author, as a “critical friend”.
The ethos is “let’s work together to make this question better”, never “this is a bad question”.
Why give and receive feedback?
In Quantum we share gifts with each other, by writing questions that others can use,
gathering questions into quizzes that others can use, and offering feedback on questions.
The challenge is this: it would be all too easy for us to assemble a large collection of notvery-good questions. One way we hope to tackle this is by analysing data on thousands of
students taking tens of thousands of questions. But another is to use the critical judgement
of our colleagues, and their experience of using our questions with their students, to improve
the questions.
Giving feedback
Here are some guidelines about giving feedback

Feedback should be constructive, and should offer help, in the form of concrete
suggestions, in addressing identified issues.
For example “You have totally misunderstood what a SSD is” is non-constructive.
Perhaps instead “Answer D is open to misinterpretation; perhaps it would be clearer
if you said XXX”.

Take time to say what you like about the question.

Never, ever be rude. The author has given up precious time to write this question,
and deserves respect.
Receiving feedback
It can be difficult to receive feedback, especially if it points out shortcomings or suggests
improvements. Authors should strive to receive critique as a gift from a busy colleague, who
has given up some precious moments to sharing their experience of reading or using the
question.
Prompts for giving feedback on questions
Here are some aspects of a question that you may want to consider in giving feedback.
Category
Examples
Spelling, punctuation,
grammar, readability
Sentence construction, use of words, etc. may
require high literacy levels
Not enough distractors
Format
Layout too crowded
Images not clear
Terminology
Misuse of vocabulary, especially subject-specific
vocabulary
Subject content
Inaccurate content
Reduction to ‘odd one out’
Design
Non-unique (infinite) number of solutions
More than one option could be correct
Unclear assessment
objective
Being unable to identify the single concept being
assessed
Need more information to complete task
Context
Need to know the context before responding
Question can be answered without the context
Options don’t make sense in relation to context
Examples
Here are some examples that have received feedback.
Example 1
Design
Reduces to ‘choose the odd one out’ because the 3 distractors are not
about a bicycle wheel
Subject
content
Abstraction should go from specific to more general
Suggestion
“Which is an abstraction of a bicycle wheel?” Provide four line drawings:
One that is clearly a bicycle wheel with spokes, one that is clearly the
entire bicycle, one that matches the bicycle wheel with teeth on the
outside like a cog, one with two concentric circles. The abstraction of a
bicycle wheel is the first. The last is an abstraction too far because it’s
not identifiable as a bicycle wheel.
Example 2
Spelling, sensibility
Missing ‘is’; hardrive should be ‘hard drive’
Terminology
The meaning of hardwired in this context is not clear. Does it
mean soldered onto the motherboard or connected with a cable?
Design
‘Motherboard’, option A has been used in the stem of the question.
Unclear assessment
objective
Is the objective to know the meaning of ‘hardwired’ or the qualities
of ROM?
Suggestion
Identify which component, found on the motherboard, stores nonvolatile instructions and data. Replace option A with ‘cache’.
Example 3
Terminology
The response to this question is dependent, in some instances,
upon what has been taught in the classroom.
Design
Although the SI prefix kilo means 1000, it has been used in
computing to mean 1024.
Context
More information is needed to be able to answer this question.
Suggested
alternative wording
Kilo is an SI unit for 1000. However, in computing, the term
kilobyte usually means (a) 8 bytes, (b) 64 bytes, (c) 1024 bytes, (d)
2048 bytes.
Example 4
Context
The distractors are unrelated to the objective of the question.
Suggestion
Change distractors (a) selection, (b) iteration, (c) global variables
Example 5
Design
Non-unique (infinite) number of solutions
A is the correct response because mathematically any derived
function could answer this question.
Suggestion
Option 1: Define what f(x) actually does in words or a
mathematical formula. Then, leave the options alone.
Option 2: Rather than ask for the next number, give four different
mathematical expressions and ask which produces all the results.
Example 6
Context
The response to this question is subjective. It would depend upon
the context of the question.
Suggestion
Provide a context: A computer scientist is collecting data by
visiting several cities around the country. Which way would you
recommend for him to back up his data?