Using Statistics in Research
Psych 231: Research
Methods in Psychology
“Generic” statistical test

Tests the question:

Are there differences between groups due to a treatment?
Two possibilities in the “real world”
H0 is true (no treatment effect)
One
population
XA
XB
Two
samples
“Generic” statistical test

Tests the question:

Are there differences between groups due to a treatment?
Two possibilities in the “real world”
H0 is true (no treatment effect)
H0 is false (is a treatment effect)
Two
populations
XA
XB
XA
XB
Two
samples
“Generic” statistical test
XA

XB
Why might the samples be different?
(What is the source of the variability between groups)?



ER: Random sampling error
ID: Individual differences (if between subjects factor)
TR: The effect of a treatment
“Generic” statistical test
XA

XB
The generic test statistic - is a ratio of sources of
variability



ER: Random sampling error
ID: Individual differences (if between subjects factor)
TR: The effect of a treatment
Observed difference
TR + ID + ER
Computed
=
=
test statistic
Difference from chance
ID + ER
“Generic” statistical test

The generic test statistic distribution

To reject the H0, you want a computed test statistics that is large
• This large difference, reflects a large Treatment Effect (TR)

What’s large enough? The alpha level gives us the decision criterion
Distribution of
the test statistic
-level determines where
these boundaries go
“Generic” statistical test

The generic test statistic distribution

To reject the H0, you want a computed test statistics that is large
• This large difference, reflects a large Treatment Effect (TR)

What’s large enough? The alpha level gives us the decision criterion
Distribution of
the test statistic
Reject H0
Fail to reject H0
“Generic” statistical test

Things that affect the computed test statistic

Size of the treatment effect
• The bigger the effect, the bigger the computed test
statistic

Difference expected by chance (sample error)
• Sample size
• Variability in the population
Some inferential statistical tests

1 factor with two groups

T-tests
• Between groups: 2-independent samples
• Within groups: Repeated measures samples (matched, related)

1 factor with more than two groups


Analysis of Variance (ANOVA) (either between groups or
repeated measures)
Multi-factorial

Factorial ANOVA
T-test

Design


2 separate experimental conditions
Degrees of freedom
• Based on the size of the sample and the kind of t-test

Observed difference
Formula:
X1 - X2
T=
Diff by chance
Computation differs for
between and within t-tests
Based on sample error
T-test

Reporting your results
The observed difference between conditions
 Kind of t-test
 Computed T-statistic
 Degrees of freedom for the test
 The “p-value” of the test
“The mean of the treatment group was 12 points higher than the
control group. An independent samples t-test yielded a significant
difference, t(24) = 5.67, p < 0.05.”
“The mean score of the post-test was 12 points higher than the pretest. A repeated measures t-test demonstrated that this difference
was significant significant, t(12) = 5.67, p < 0.05.”



Analysis of Variance

Designs

XA
XB
XC
More than two groups
• 1 Factor ANOVA, Factorial ANOVA
• Both Within and Between Groups Factors


Test statistic is an F-ratio
Degrees of freedom


Several to keep track of
The number of them depends on the design
Analysis of Variance
XA

XB
XC
More than two groups


Now we can’t just compute a simple difference score since
there are more than one difference
So we use variance instead of simply the difference
• Variance is essentially an average difference
Observed variance
F-ratio =
Variance from chance
1 factor ANOVA
XA

XB
XC
1 Factor, with more than two levels

Now we can’t just compute a simple difference score since
there are more than one difference
• A - B, B - C, & A - C
1 factor ANOVA
Null hypothesis:
XA
XB
H0: all the groups are equal
XA = XB = XC
Alternative hypotheses
HA: not all the groups are equal
XA ≠ XB ≠ XC
XA = XB ≠ XC
XC
The ANOVA
tests this one!!
Do further tests to
pick between these
XA ≠ XB = XC
XA = XC ≠ XB
1 factor ANOVA
Planned contrasts and post-hoc tests:
- Further tests used to rule out the different
Alternative hypotheses
XA ≠ XB ≠ XC
Test 1: A ≠ B
Test 2: A ≠ C
Test 3: B = C
XA = XB ≠ XC
XA ≠ XB = XC
XA = XC ≠ XB
1 factor ANOVA

Reporting your results
The observed differences
 Kind of test
 Computed F-ratio
 Degrees of freedom for the test
 The “p-value” of the test
 Any post-hoc or planned comparison results
“The mean score of Group A was 12, Group B was 25, and
Group C was 27. A 1-way ANOVA was conducted and the
results yielded a significant difference, F(2,25) = 5.67, p < 0.05.
Post hoc tests revealed that the differences between groups A
and B and A and C were statistically reliable (respectively t(1) =
5.67, p < 0.05 & t(1) = 6.02, p <0.05). Groups B and C did not
differ significantly from one another”


Factorial ANOVAs


We covered much of this in our experimental design lecture
More than one factor



Factors may be within or between
Overall design may be entirely within, entirely between, or mixed
Many F-ratios may be computed


An F-ratio is computed to test the main effect of each factor
An F-ratio is computed to test each of the potential interactions
between the factors
Factorial ANOVAs

Reporting your results

The observed differences
• Because there may be a lot of these, may present them in a table
instead of directly in the text

Kind of design
• e.g. “2 x 2 completely between factorial design”

Computed F-ratios
• May see separate paragraphs for each factor, and for interactions

Degrees of freedom for the test
• Each F-ratio will have its own set of df’s

The “p-value” of the test
• May want to just say “all tests were tested with an alpha level of
0.05”

Any post-hoc or planned comparison results
• Typically only the theoretically interesting comparisons are
presented