SWANSEA UNIVERSITY
Response Pads
The design, development and results of a
bespoke response box, and subsequent testing,
for comparison purposes, of various devices
used for reaction time measurements in
psychological experiments
by
Gary Freegard
Department of Psychology
20/8/2012
Psychology Response Box
This report supersedes the previous report (dated 1/3/12), it contains a brief history of the design
and development of the Response Box, a comparison of the Response Box against various other
input devices currently used in reaction time measurement in the department and 2 methods to
improve the recorded accuracy. Also to improve the accuracy and reliability of the data a larger
sample was used (normally 10,000 measurements up from 1,000).
This Response Box is shown to have a significantly greater accuracy and precision than all other
options currently available in the Psychology Department (see Conclusion page 9).
Latency is the measure of the time delay experienced in a system. In this report it is the time taken
by a PC to respond after a button has been pressed on a response box or keyboard.
Accuracy is the closeness of measurements to the actual value, the average of the results.
Precision is the repeatability of the measurements, the standard deviation of the results.
All measurements are in millisecond (ms), the vertical axis of all the graphs show the response time
which is the time from button/key press to the output from the parallel port.
The packages used for the testing are as follows:
E-Prime
VB.net
Opensesame
Most of the graphs used here have a vertical axis range from 0 to 40 ms for ease of comparison,
unless otherwise stated.
© Gary Freegard 2012
1
Design History
Purpose
To design and build a response box that fulfils the following criteria:
use USB standard for communication
‘Plug and Play’ on multiple operating systems (OS)
lower latency than a standard keyboard (8-12ms, Cedrus website)
functionally comparable with our current response boxes (Cedrus)
design limited only by customer specification and by available parts
upgradeable/expandable
Design
To fulfil the design requirements, a micro-controller was required, along with custom software.
Currently, a micro-controller development system called Flowcode is being used in the department
for the Microchip Pic range of controllers; this limits the list of controllers available for the design.
The possible methods of communication are either USB serial (as current units) or USB HID
(Human Interface Device) i.e. keyboard, mouse, joystick etc. The USB HID requires no new drivers
(though the operating system will see it as a new device).
The USB Serial offers two options:
1, to use the micro-controller in USB Serial mode (but availability of drivers for the other
operating systems is a possible limitation)
2, use a separate USB serial device (made by FTDI) which is also used in the Cedrus boxes
(drivers are available for all operating systems)
The USB HID option was chosen as there is no uncertainty with regards drivers.
The micro-controller initially selected was the ECIO (18F4455 Microchip Pic) designed and built
by Matrix Multimedia, the makers of Flowcode. This device is a 40 pin DIL (Dual In Line) with a
USB port attached and with a boot-loader installed. This allows the device to be reprogrammed via
its own USB port and therefore doesn’t need to be removed from the circuit, which aids
development significantly. Since this device is ready to run out of the box, the only other parts
required to fulfil the design specification are the resistors, switches and case.
Response Box Mk1
This was built for proof of concept, and software development.
This consisted of a metal unit with four buttons on the top panel for participant reaction time
measurements plus two buttons on the rear, one to reset the unit to allow reprogramming and the
other to output a ‘\’ for the Mac OS if required. This used only 6 of the 30 available IOs
(Inputs/Outputs).
figure 1
© Gary Freegard 2012
2
This unit works on Windows, Mac OS and Linux without the need for additional drivers. The
latency was 2.4ms at worst, verified by the results shown in figure 12 and table 1 (page 7).
Response Box Mk2
This version was built to meet a specific requirement of an academic staff member. It has 5
response buttons along the top (1 to 5), 3 along the bottom (a, b and c) and it has 5 buttons in the
centre, 4 of which are arrow keys, these have the same effect as the arrow keys on the keyboard,
and the fifth is an enter button.
figure 2
Unfortunately the physical size of the ECIO made it unsuitable for this new design, so instead a
18F2455 was used (a 28 pin device with inbuilt USB capabilities with 19 IO pins and
approximately a third of the price). A printed circuit board was designed and built in house.
figure 3
Using this micro-controller would allow the design of response boxes with up to 90 buttons if so
desired, whilst if it bigger brother (18F4455) was used then it would be possible to create response
box with 160 buttons!
© Gary Freegard 2012
3
The Competition:
RB730- part of the RBx30 series of response boxes from Cedrus, makers of SuperLab
figure 4
There are 4 different models in this range, with varying layout and number of keys, but they
all use the same underlying hardware.
They are limited to a maximum of 8 keys.
They are fitted with an accessory connector that provides 6 configurable inputs/outputs.
They are compatible with the majority of stimulus presentation software.
The website also claims that their pad offers a 1 millisecond reaction time resolution.
SR200A by PST, makers of E-Prime
figure 5
This box has only 5 buttons and 5 lamps that can be software controlled.
It also has a Microphone input with variable gain and trigger level.
It has an inbuilt Screen refresh detector circuit which uses the optional PST Refresh
Detector.
It offers 0 millisecond debounce period
Uses the serial port though a Serial to USB converter can be used.
This unit is compatible only with software specifically written to use this unit.
Mac Keyboard
figure 6
Dell Keyboard
figure 7
© Gary Freegard 2012
4
Mouse
figure 8
IntelliMouse by Microsoft, uses USB communication.
© Gary Freegard 2012
5
Testing the Response Boxes
A program was written in VB.net (initially and then other software packages) to respond to button
presses. Once a button press has been received by the Test PC, the test program will produce a 10
millisecond pulse on the parallel port (printer port), the stop signal. Initially a counter timer was
used to measure the time between the button being pressed and the pulse from the parallel port.
There is a latency of approximately 0.050ms (the max is 0.13ms see tables 30/31 page 24) between
the response of the program to a button pressed (on response box) and the port going high.
Initial testing was carried out using the setup shown in figure 9 below. This is a purely manual
approach, the button was pressed manually and the time data was recorded manually.
figure 9
After several hundred key presses a secondary 18F2455 was programmed to measure and output
(via USB Serial) the times between two events (pulses) on separate inputs. This enabled the
collection of larger sample of timing data and also facilitated the automation of the process (see
figure 10 page 7)
© Gary Freegard 2012
6
The Timer Unit is able to carry out repeated measurements in microseconds (a 1000 microseconds
equals 1 millisecond) and is calibrated to an accuracy of 0.02% + 1 l.s.d.(least significant digit) by
using an external counter timer. The data is sent to a PC by USB Serial and is captured by
HyperTerminal or similar.
Below is a diagram showing the setup for automatic testing, to reduce unnecessary load on the test
PC all timing data was collected by a second PC.
figure 10
Here is a breakdown of the operation involved in collecting the timing data:
The Press Button signal (sent from the Timer Unit) is used to simulate the pressing of a
button/key in the Response Box/Keyboard
At the same time the Timer Unit internal clock is started
The Response Box/Keyboard sends the button/key data to the Test PC
Upon receipt of the button/key data the Test PC sends a Stop signal via the Parallel Port to
the Timer Unit.
The Timer Unit stops its internal clock and the measured time is sent to the Data PC for
collection
After a random delay (20 to 270ms), the Press Button signal is sent and the process starts
again.
© Gary Freegard 2012
7
Opensesame and VB.net programs required the installation of inpout32.dll to enable the outputting
of data via the parallel port (printer port). Whilst E-Prime comes with a port option built in.
The test programs for Opensesame and E-Prime.
figure 11
figure 12
The graph (figure 13) and table 1 below shows the data collected during the testing of the Response
Box using the USB HID interface (keyboard) with a program written in VB.net. The vertical axis (0
to 40 milliseconds) shows the response time of the device under test whilst the horizontal axis
displays the number of key presses (real or simulated).
Min
1.122
Max
2.283
Average
1.671
StDev
0.289
table 1
figure 13
This graph shows the Time (Response) in milliseconds for 10000 button presses. The results show
the difference between the actual time (zero) and the time taken for the PC to respond to a button
press. The zoomed in area of the graph shows the response times for the first 100 presses, with lines
indicating the average (red), plus 1 standard deviation (green) and minus 1 standard deviation
(purple).
The table (table 1) shows the statistical data for the data shown in figure 13.
From table 1 it can be seen the average (accuracy) is 1.671ms, whilst the standard deviation
(precision) is 0.289ms.
The minimum and maximum delays are 1.122ms and 2.283 respectively.
The ideal response box would have an average of 0ms and a standard deviation of 0ms.
© Gary Freegard 2012
8
Conclusion
Over the next three pages are the conclusions/comparisons of the testing of all the devices, using
three different applications.
All these tests were carried out on a Dell Optiplex 755 (3GHz Duo Core) with Windows XP Sp3
and a laptop to record the timing data.
The compatibility table below shows the device and the application that it was tested with. No
testing of the RB730 or SR200A was carried out with the VB.net application, as I was unaware of
the protocol needed to use these devices.
Compatibility
VB.net E-Prime
Response Box USB Hid


Response Box USB Serial


Response Box FTDI Serial


Response Box MAX232 Serial
1
1
Cedrus RB730


SR200A


Mac Keyboard


Dell Keyboard


Mouse


table 2
Opensesame

2
2
2





Notes:
1- Was unable to test automatically as button presses were missed
either by the PC or software only 300 presses recorded
2-Opensesame does not come with a serial port option.
© Gary Freegard 2012
9
E-Prime
The Response Box achieves accuracy and precision that is significantly better than all the other
current response units, some of which come from specialist manufactures.
figure 14
Total Time Data in ms
Average
Stdev
Response
Box
2.503
SRB200
6.970
0.333
3.181
Mac
Keyboard
8.250
Cedrus
RB-x30
10.977
2.598
table 3
4.643
Mouse
18.415
Dell
Keyboard
27.264
2.372
3.168
Comparison against Response Box
SRB200
Average
2.780
StDev
9.520
Mac
Keyboard
3.290
Cedrus
RB-530
4.380
7.750
13.900
table 4
Mouse
7.350
Dell
Keyboard
10.980
7.100
9.490
From this table it can be seen that the Response Box’s accuracy and precision compared to its
nearest rival (SRB200) are 2.78 and 9.52 times better, respectively.
These are the results of the tests on pages 13, 17, 18, 19, 20 and 21.
© Gary Freegard 2012
10
VB.net
Again the response box achieves an accuracy and precision that is significantly better than the two
other units that are compatible with the VB.net program.
figure 15
Average
StDev
Response
Box
1.671
Mac
Keyboard
7.629
0.289
2.560
table 5
Mouse
17.774
Dell
Keyboard
26.531
2.315
3.114
The nearest rival is the Mac Keyboard, which is 4.5 times less accurate and 8.9 times less precise.
Comparison against Response Box
Accuracy
Precision
Mac
Keyboard
4.570
Mouse
8.860
table 6
10.640
Dell
Keyboard
15.880
8.010
10.770
These are the results of the tests on pages 13, 19, 20 and 21.
© Gary Freegard 2012
11
Opensesame
And again the Response Box comes out on top when compared with the competition.
figure 16
Average
StDev
Response
Box
2.105
Mac
Keyboard
8.013
0.296
2.611
RBx30
Mouse
11.594
18.18
Dell
Keyboard
26.762
4.701
table 7
2.339
3.112
SR200A
6.168
2.936
Once again the nearest rival is the SR200A, which is 2.9 times less accurate and 9.9 times less
precise.
Comparison against Response Box
SR200A
Average
2.931
Mac
Keyboard
3.807
StDev
9.905
8.810
RBx30
Mouse
5.509
8.638
Dell
Keyboard
12.716
15.858
7.891
10.498
table 8
These are the results of the tests on pages 14, 17, 18, 19, 20 and 21.
© Gary Freegard 2012
12
Response Box Results
A script was written in E-Prime to output a 10ms pulse on the parallel port once a keyboard/button
press was detected. The results are below; this is visibly different to the results from either of the
Keyboard and Serial as the standard deviation is less, the minimum is 0.7ms higher and maximum
is 2ms higher.
Min
1.839
Max
4.372
Average
2.503
StDev
0.333
table 9
figure 17
This shows that the Response Box works with E-Prime and that the times recorded are comparable
with those from VB.net, though slightly higher but with less variation. From these results it can be
seen that there is a fixed latency of approx 1.8ms and this could be subtracted from all the recorded
reaction times.
figure 18
© Gary Freegard 2012
Min
1.122
Max
2.283
Average
1.671
StDev
0.289
table 10
13
figure 19
Min
1.526
Max
3.722
Average
2.105
StDev
0.296
table 11
Unfortunately E-Prime does not allow the use of a USB or Normal Serial device as an input device
unless it emulates a PST (E-Prime manufacturer) response box, so the only way to carry out any
tests using Serial mode was by the use of an inline script that would wait for data to be received
before sending a Stop signal to the Timer unit.
With Opensesame it was not possible to test the Serial mode out,
Min
2.031
Max
7.162
Average
2.717
StDev
0.316
table 12
figure 20
Min
1.13
Max
2.235
Average
1.684
StDev
0.291
table 13
figure 21
© Gary Freegard 2012
14
The following tests were carried out to check the methods of communication used by Cedrus and
PST in their response units, by attaching these devices to the Response Box. This required the
modification of the Response Box’s software so that the keypresses were output as Serial data,
which is then sent to the attached device (either FTDI or MAX232).
Min
1.799
Max
33.987
Average
10.455
StDev
4.691
table 14
figure 22
Min
1.114
Max
17.692
Average
9.212
StDev
4.659
table 15
figure 23
This data is similar to the data collected for the Cedrus unit (figure 27 and table 19 on page 17), the
Average and Standard Deviation are very close, indicating similar level of performance.
© Gary Freegard 2012
15
From the below graph and table it can be seen that using the old RS232 comport would allow
devices to be very precise, this method has the lowest standard deviation of any of the tests that
have been carried out. Unfortunately this port is no longer standard fit on computers, due to it being
superceeded by USB, though it is possible to purchase PCI/PCI-E expansion cards with RS232
ports.
During testing it was not possible to run it automatically, as not all the button presses were received
by the PC, this hi-lights an error in the RS232 hardware/setup, as the Response Box was using the
same software as used in the FTDI test (page 15).
Min
3.696
Max
4.337
Average
3.757
StDev
0.077
table 16
figure 24
Min
3.827
Max
7.501
Average
3.971
StDev
0.279
table 17
figure 25
© Gary Freegard 2012
16
Cedrus RB-x30 response box results
This device uses the FTDI for its USB Serial communications. It was tested to check if the results
from Response Box were comparable to this device.
Min
1.777
Max
19.84
Average
10.977
StDev
4.643
table 18
figure 26
The results show that the Response Box is more accurate (5x) and has a higher precision (22x) than
the Cedrus unit when used with E-Prime
This result was totally unexpected, as the belief was that these Cedrus units do record reaction times
accurately and the intention was to benchmark Response Box against the Cedrus.
Min
2.36
Max
20.588
Average
11.593
StDev
4.700
table 19
figure 27
© Gary Freegard 2012
17
SRB200A Response Box
This box is designed by PST specifically to be used with E-Prime. It is a system that was designed
in 1997 is limited to 5 buttons internally (8 buttons via the accessory socket) and requires a serial
port. Though it does have a Microphone input and a Refresh Detector System built in.
Min
1.236
Max
16.523
Average
figure 28
6.970
StDev
3.181
table 20
This unit achieves better results than the Cedrus unit but it still falls short of the results achieved
with the in house designed unit (Response Box) which has 3.5 times better accuracy and the
precision is 14 times better.
Min
1.082
Max
11.438
Average
figure 29
6.168
StDev
2.936
table 21
Another surprise is that a better result for both accuracy and precision is achieved using
Opensesame.
© Gary Freegard 2012
18
Mac Keyboard
This is a standard Mac keyboard. During the testing this was the surprise device, it was significantly
better than a standard USB keyboard and was better than the RBx30 device.
Min
1.985
Max
14.255
Average
figure 30
Min
1.703
Max
13.627
Average
figure 31
© Gary Freegard 2012
7.629
StDev
2.560
table 23
Min
2.046
Max
14.055
Average
figure 32
8.250
StDev
2.598
table 22
8.013
StDev
2.611
table 24
19
Mouse
For Windows the polling rate is set automatically to 125Hz, so the expected latency should be 8ms
but as you can see the average latency is approximately 18ms. So this polling rate may only apply
to mouse movement and not the buttons.
The graphs and statistics below show that the mouse’s performance is almost identical for all
software packages.
The only positive point about using this mouse is that it out performs the Dell keyboard.
Min
12.811
Max
23.206
Average
18.415
StDev
2.372
table 25
figure 33
Min
13.218
Max
22.408
Average
17.774
StDev
2.315
table 26
figure 34
Min
13.542
Max
22.855
Average
18.180
StDev
figure 35
© Gary Freegard 2012
2.339
table 27
20
Standard Keyboard Results
The purpose built timer was connected to a keyboard button and again to the parallel port. Below
are the results, as can be seen the results are significantly worse than was expected (8-12ms).
Min
17.542
Max
33.493
Average
26.531
StDev
table 28
figure 36
figure 37
3.114
Min
18.303
Max
34.717
Average
27.264
StDev
3.168
table 29
Min
17.936
Max
34.091
Average
26.762
StDev
figure 38
3.112
table 30
The Response Box accuracy is at least 10 times better and its precision is at least 9 times better than
this keyboard. These results are far worse than was expected (8-12ms) and these keyboards should
only be used when accurate reaction times are not needed across any presentation software.
© Gary Freegard 2012
21
Additional Information
System Latencies
The latencies are caused by the areas shown in the figure 45 (below), these are:
Red– the time it takes the microcontroller to respond to a key press and then send an output
via either the serial or USB interface and then for this interface to transmit this data on to the
PC. Delays here can be caused by using debounce routines and/or keyboard scan routines.
Green– the time for the Input/Output ports and associated hardware to respond to the
incoming data, this is dependent on the method used to acknowledge/transfer the receipt of
data to the operating system; an example is the USB system where data is only transferred
once every 1 millisecond.
Blue – the time for the operating system to pass the data onto the application and then for
this application to respond and then to initiate the stop signal. This is dependent on the
operating system being used (MacOS, Linux or Windows), the speed of the processor,
number of cores, workload etc.
Yellow-the time for the PC hardware to produce an output on the parallel port.
figure 39
© Gary Freegard 2012
22
PC Port Test
The same method (as used to capture all the Total Response Times) was used to measure the latency
caused by the PC and parallel port (figure 41 below). The Timer Unit (for connections see figure 40
below) was used to produce a Start pulse into the input on the parallel port at the same time its
internal clock was started. Two applications were written in VB.net and E-Prime to respond to an
input on the parallel port by producing an output also on the parallel port, the Stop signal. The
Timer Unit stopped its clock on receipt of the Stop signal. This was done for 1000 trials and the
results are in tables 31/32, and it can be seen that the worst case latency is 0.133ms.
E_Prime
VB.net
Response Time ms
Response Time ms
Min
0.031
Min
0.023
Max
0.133
Max
0.109
Average
0.048
Average
0.049
0.008
StDev
StDev
table 31
figure 40
© Gary Freegard 2012
0.008
table 32
figure 41
23
USB HID
figure 42
Explanation of USB HID timing diagram (figure 42)
Upon a button being pressed, the micro-controller attempts to send the data to the PC but this isn’t
possible until the micro-controller receives a USB Polling signal (every 1ms). This is the cause of
the first latency (referred to as Initial Delay in this report) which can vary from 0.1 to 1.3 ms (see
table 13 page 12 : min-1ms and max-1ms).
Once this button data has been sent, the micro-controller has to send an empty data set upon
receiving the next polling signal (1ms later). It is upon receipt of this that the PC will respond to the
button press i.e. the application will receive button data.
The total system latency is made up of the following:
Initial Delay
Sending the empty data
Driving the parallel port
The Initial Delay is variable, but has a direct correlation with the Total Response Time.
The sending of the empty data set is a fixed delay of 1ms
The parallel port is 0.05ms on average (see table 31/32page 23).
So if all the delays are know then it will be possible to adjust the recorded reaction time to the
actual reaction time.
© Gary Freegard 2012
24
Useful Links
18F2455 and 18F4455
http://ww1.microchip.com/downloads/en/DeviceDoc/39632e.pdf
ECIO
http://www.matrixmultimedia.com/ecio.php
Flowcode
http://www.matrixmultimedia.com/flowcode.php
USB
http://www.beyondlogic.org/usbnutshell/usb1.shtml
http://www.usb.org/developers/devclass_docs/HID1_11.pdf
http://en.wikipedia.org/wiki/Universal_Serial_Bus
http://www.usbmadesimple.co.uk/index.html
Cedrus
http://www.cedrus.com/
http://www.cedrus.com/responsepads/why_use.htm)
E-Prime
http://www.pstnet.com/
http://www.pstnet.com/support/kb.asp?TopicID=1835
Empirisoft
http://www.empirisoft.com/Default.aspx?index=0
http://www.empirisoft.com/Hardware.aspx
Opensesame
http://www.cogsci.nl/software/opensesame/
FTDI
http://www.ftdichip.com/
PIC and PICmicro are registered trademarks of Arizona Microchip Inc.
Flowcode and E-Blocks are trademarks of Matrix Multimedia Limited
E-Prime is a registered trademark of Psychology Software Tools, Inc.
Windows is a registered trademark of Microsoft Corporation.
Mac and Mac OS are trademarks of Apple Inc.
© Gary Freegard 2012
25