Experiments
Chapter 2 – Review
• Types of behavioral research (descriptive,
relational, experimental)
• Research hypotheses
• Basics of experimental research
9/22/2016
2
Variables
Independent variables (IV) refer to the factors that the researchers are
interested in studying or the possible “cause” of the change in the
dependent variable.
Dependent variables (DV) refer to the outcome or effect that the
researchers are interested in and measuring.
Speed of selecting commands with mouse and keyboard commands
9/22/2016
3
Research hypotheses
• A hypothesis is a precise (falsifiable) problem statement that
can be directly tested through an empirical investigation.
• It is a supposition or proposed explanation made on the basis
of limited evidence as a starting point for further investigation
A provisional answer to a research question.
• Has to define the variables involved
• Has to define a relationship
No difference in performance for mouse and keyboard when
selecting a command
9/22/2016
4
Components of experiment
• Treatments, or conditions: the different techniques,
devices, or procedures that we want to compare
• Units: the objects to which we apply the experiment
treatments. In HCI research, the units are normally
human subjects with specific characteristics, such as
gender, age, or computing experience
• Assignment method: the way in which the experimental
units are assigned different treatments.
9/22/2016
5
Experiment -practice
Video (stop at 1:19), design the experiment
• Lady Tasting Tea - Inferential Statistics and
Experimental Design https://www.youtube.com/watch?v=lgs7d5saFFc
9/22/2016
6
Overview, Ch 3, Design of experiments
• What needs to be considered when designing
experiments?
• Determining the basic design structure
• Investigating a single independent variable
• Investigating more than one independent variable
• Reliability of experimental results
• Experimental procedures
9/22/2016
7
Three groups of studies
• Experiments, quasi-experiments, and nonexperiments:
9/22/2016
8
True or quasi experiment?
9/22/2016
9
Characteristics of true experiments
• A true experiment
– is normally based on at least one hypothesis
– have multiple conditions
– The dependent variable can be quantitatively
measured
– uses statistical significance tests
– thrives to remove biases
– is replicable
9/22/2016
10
Factors to consider
• Research hypothesis
– Clearly defined
– Appropriate scope
• Dependent variables
– Easy to measure
• Independent variables and conditions
– Easy to control
9/22/2016
11
Basic design structure
• Two basic questions:
– How many independent variables do we want to
investigate in the experiment?
– How many different values does each
independent variable have?
9/22/2016
12
Basic design structure
9/22/2016
13
Examples
• H1: No differences in typing speed when using
a QWERTY keyboard, a DVORAC keyboard or
an alphabetically ordered keyboard
• H2: No difference in trust to robots among
elderly in Japan and Norway
• H3: No difference in typing speed among
novices and experts
9/22/2016
14
Investigating one independent variable Between group design
• Also called ‘between subject design’
• One participant only experience one condition
9/22/2016
15
Between group design
• Advantages
– Cleaner, better control of learning effect
– Requires shorter time for participants
– less impact of fatigue and frustration
• Disadvantages
– Impact of individuals difference
– Harder to detect difference between conditions
– Require larger sample size
9/22/2016
16
Investigating one independent variable Within group design
• Also called ‘within subject design’
• One participant experience multiple
conditions
9/22/2016
17
Within-group design
• Advantages
– Requires smaller sample size
– Easy to detect difference between conditions
• Disadvantages
– Learning effect
– Takes longer time
– Larger impact of fatigue and frustration
9/22/2016
18
Investigating one independent variable Between group vs. Within group
• Between-group design should be taken when:
– Simple tasks
– Learning effect has large impact
– Within-group design is impossible
• Within-group design should be taken when:
– Learning effect has small impact (different fonts,
reading speed)
– Small participant pool (experts)
9/22/2016
19
How to improve within-group design
• Learning effect
– Randomize the order of the experimental
conditions (randomization software)
– Training
• Fatigue
– Sessions 60-90 minutes
– Breaks (forced breaks)
9/22/2016
20
More than one independent variable
• Factorial design divides the experiment groups or
conditions into multiple subsets according to the
independent variables
• Can study interaction effects
• Number of conditions:
• Three types of keyboards, two type of tasks
(transcription, composition) 3 x 2 = 6 conditions
9/22/2016
21
More than one independent variable
• Three options of factorial design
– Between group design
– Within group design
– Split-plot design
• Split-plot design
– Has both a between-group and a within-group component
9/22/2016
22
Keyboard example
• Between-group
– each participant completes task under only one
condition
– We need 6 groups one for each condition
• Within-group
– Each participant completes task under all six
conditions
9/22/2016
23
Split-plot
• Independent variables: age and use if GPS
when driving
• Number of conditions is 6
• Combination of between-group and within
group
• Users divided in three groups (age) and each
user asked to drive with and without GPS
9/22/2016
24
Interaction effect
• The differing effect of one independent variable on the
dependent variable, depending on the particular level of
another independent variable
9/22/2016
25
Reliability of experiments
• Random errors
–
–
–
–
Also called ‘chance errors’ or ‘noises’
Cause variations in both direction
Occur by chance
Can be controlled by a large sample size
• Systematic errors
– Also called ‘biases’
– Always push actual value in the same direction
– Can never be offset no matter how large the sample is
9/22/2016
26
Reliability of experiment results
• Five major sources of system errors
– measurement instruments
– experimental procedures
– participants
– experimenter behavior
– the experimental environment
9/22/2016
27
Lifecycle of an experiment
• Identify a research hypothesis
• Specify the design of the study
• Run a pilot study to test the design, the system, and
the study instruments
• Recruit participants
• Run the actual data collection sessions
• Analyze the data
• Report the results
9/22/2016
28
Experiment session procedure
• Ensure the systems or devices being evaluated
and the related instruments are ready for the
experiment
• Greet the participants
• Introduce the purpose of the study and the
procedures
• Get the consent of the participants
• Assign the participants to a specific
experiment condition according to the predefined randomization method
9/22/2016
29
Experiment session procedure
•
•
•
•
•
Participants complete training task
Participants complete actual tasks
Participants answer questionnaires (if any)
Debriefing session
Payment (if any)
9/22/2016
30
Measurements
–
–
–
–
–
–
–
–
Task performance
Time performance
User satisfaction (validated survey tool)
Average time to recover from an error
Time spent using help
Number of visits to the search feature
Time spent on specific web page
Typing speed
9/22/2016
31
Starten av dagen:
• Sjekke at alt er på plass i laben; gavekort, utstyr, godteri, brus, batterier, pen og papir osv.
• Skrive ut eksperimentet material som skal deles til deltagerne
• Skrive ut tlf.nr. til de som kommer (mail fra Nordstat)
• Informere resepsjonen at vi venter deltakere. Hvis deltaker kommer veldig tidlig, be resepsjonisten
henvise dem til sofa i resepsjonen.
• Logge seg på PC og teste alt virker (ingen updates skal dukke opp i løpet av eksperimentet)
• Gå inn på facebook og twitter og sjekk at ”xxxx er logget ut
• Heng IKKE FORSTYR på døra
• Ha alltid mobilen med deg
Eksperimentet (før deltakeren kommer)
• Start SINTEF verktøyet (r2d2exp.jar). Det kan ta litt tid – IKKE KLIKK FLERE GANGER!
• Start opp camtasia studio (dobbelt klikk)
• Start opp uLog (høyre klikk på ikonet i ”system tray” og start)
• Eksperiment (hent deltaker)
• Ta med deltageren ned; hilse; introduksjon
• Konfidensialitet skjema signeres
• Husk å sette mobilen på vibrasjon
• Start opptak i camtasia studio (høyreklikk, klikk ‘record the screen’, klikk record button)
• …
9/22/2016
32
9/22/2016
33
Exercise
• Groups 4-5 students
• You want to make a tool helping elderly (55+) to
remember taking their medicine. What works
better – a robot , a smart watch, a smart pillbox?
Does it work differently for different groups (man,
woman, <70, 70+, 90+…) How to design an
experiment? What are the possible
problems/limitations?
• 10 minutes
9/22/2016
34