Advanced experimental design and analysis tes
Advanced
Experimental Design
Topic 6
Chapters 11 & 12
Oneway Analysis of Variance
Multiple Comparisons
1
Agenda
Analysis of Variance (with
computational illustration)
| Comparison of F and t, connection
| Magnitude of effect (η2, ω2)
| Power
| Multiple Comparisons
|
2
Multiple groups (>2)
What if we have a situation where we
have more than two means that we
wish to compare with one another?
| How would we approach assessing
mean differences?
| Situations this might be an issue?
|
3
1
Type I error rates
could use multiple t-tests, but why might that
not be a good idea?
| inflating the probability of making a type I error
| Error rate per experiment
|
z
z
sum the p values for all of the tests
expected rate of type 1 errors = 6 x 0.05 = 0.30.
4
Three or more groups
Common in outcomes research,
where we may wish to evaluate the
relative efficacy of a number of
different treatments
| Few studies we might be interested in
doing have only two groups
| Oneway ANOVA allows comparisons
among two or more sample means
| Examples of multiple group studies
|
5
ANOVA Structural Model
X ij = µ + τ j + ε ij
τ j = (µ j − µ )
i=# of people, j=# of groups, Xij is the
ith person in the jth group
| µ = grand mean
| τ = adjustment reflecting mean of
jth group
| ε = “uniqueness”
|
6
2
ANOVA: ANalysis Of VAriance
|
Assumptions - with a large enough sample
can relax the assumptions
z
z
z
|
normal distribution of dependent variable
equal variances of groups
Independence of observations
Considered to be ROBUST
z
especially insensitive to violations of
normality when equal numbers in each
group and the distributions are about the
same shape
7
ANOVA conceptualization
Variance in a set of observations can be
partitioned into within groups and between
groups variance
| F statistic: basically the ratio of the between
groups variance to within groups variance
| If no differences between the groups means,
the between and within groups variance will be
about the same, resulting in an f ratio of
around one
|
8
ANOVA degrees of freedom
|
|
|
|
F statistic has two degrees of freedom:
between groups df which = ngroups-1
within groups df which = n - ngroups
F (2, 45) = 6.16, p < .01
ANOVA
CORRECT NUMBER OF STEERING CORRECTIONS
Between Groups
Within Groups
Total
Sum of
Squares
1016.667
3711.250
4727.917
df
2
45
47
Mean Square
508.333
82.472
F
6.164
Sig.
.004
9
3
Steps involved in hypothesis
testing w/ANOVA
1.
2.
3.
4.
5.
6.
7.
Identify H0 and H1
Tentatively assume H0
Choose a sampling distribution
Obtain data, calculate sample
statistic
Compare Fobs with Fcrit
Reach conclusion re: status of H0
Tell the story in terms of constructs
10
ANOVA example
hired by a large corporation to
implement a “wellness program.”
| encouraging good dietary habits and
exercise will decrease sick days,
productivity will increase and have
reduced health insurance benefit costs
due to lower utilization of medical
services
| best way to present the program, in
terms of scheduling?
|
11
ANOVA example (cont.)
freedom to schedule any time you wish
| attendance will be mandatory
| want to find out what people will prefer
| more satisfied with the program = receptive and
attentive to the information presented
| group of 30 employees randomly assigned to
each of the three conditions
| at the conclusion of the program (identical for all
3 groups) assess satisfaction with the program
|
12
4
The data
Condition Satisfaction Cond. Satis. Cond.
Satis.
days
5
nights
5 Saturday
5
days
5
nights
4 Saturday
5
days
5
nights
4 Saturday
5
days
5
nights
3 Saturday
4
days
4
nights
3 Saturday
4
days
4
nights
2 Saturday
4
days
4
nights
2 Saturday
3
days
4
nights
2 Saturday
3
days
3
nights
1 Saturday
3
days
2
nights
0 Saturday
2
13
Calculating F
1681 + 676 + 1444
10
2(df )
X days
5
5
5
5
1052
4
−
4
30 = 6.3
4
4
3
2
41
ΣX
2
1681
(ΣX)
Variance
0.99
Mean
4.1
Average variance
MSw
1.44
f=MSb/MSw=
fcrit(2,27) p
Experimental Design
Topic 6
Chapters 11 & 12
Oneway Analysis of Variance
Multiple Comparisons
1
Agenda
Analysis of Variance (with
computational illustration)
| Comparison of F and t, connection
| Magnitude of effect (η2, ω2)
| Power
| Multiple Comparisons
|
2
Multiple groups (>2)
What if we have a situation where we
have more than two means that we
wish to compare with one another?
| How would we approach assessing
mean differences?
| Situations this might be an issue?
|
3
1
Type I error rates
could use multiple t-tests, but why might that
not be a good idea?
| inflating the probability of making a type I error
| Error rate per experiment
|
z
z
sum the p values for all of the tests
expected rate of type 1 errors = 6 x 0.05 = 0.30.
4
Three or more groups
Common in outcomes research,
where we may wish to evaluate the
relative efficacy of a number of
different treatments
| Few studies we might be interested in
doing have only two groups
| Oneway ANOVA allows comparisons
among two or more sample means
| Examples of multiple group studies
|
5
ANOVA Structural Model
X ij = µ + τ j + ε ij
τ j = (µ j − µ )
i=# of people, j=# of groups, Xij is the
ith person in the jth group
| µ = grand mean
| τ = adjustment reflecting mean of
jth group
| ε = “uniqueness”
|
6
2
ANOVA: ANalysis Of VAriance
|
Assumptions - with a large enough sample
can relax the assumptions
z
z
z
|
normal distribution of dependent variable
equal variances of groups
Independence of observations
Considered to be ROBUST
z
especially insensitive to violations of
normality when equal numbers in each
group and the distributions are about the
same shape
7
ANOVA conceptualization
Variance in a set of observations can be
partitioned into within groups and between
groups variance
| F statistic: basically the ratio of the between
groups variance to within groups variance
| If no differences between the groups means,
the between and within groups variance will be
about the same, resulting in an f ratio of
around one
|
8
ANOVA degrees of freedom
|
|
|
|
F statistic has two degrees of freedom:
between groups df which = ngroups-1
within groups df which = n - ngroups
F (2, 45) = 6.16, p < .01
ANOVA
CORRECT NUMBER OF STEERING CORRECTIONS
Between Groups
Within Groups
Total
Sum of
Squares
1016.667
3711.250
4727.917
df
2
45
47
Mean Square
508.333
82.472
F
6.164
Sig.
.004
9
3
Steps involved in hypothesis
testing w/ANOVA
1.
2.
3.
4.
5.
6.
7.
Identify H0 and H1
Tentatively assume H0
Choose a sampling distribution
Obtain data, calculate sample
statistic
Compare Fobs with Fcrit
Reach conclusion re: status of H0
Tell the story in terms of constructs
10
ANOVA example
hired by a large corporation to
implement a “wellness program.”
| encouraging good dietary habits and
exercise will decrease sick days,
productivity will increase and have
reduced health insurance benefit costs
due to lower utilization of medical
services
| best way to present the program, in
terms of scheduling?
|
11
ANOVA example (cont.)
freedom to schedule any time you wish
| attendance will be mandatory
| want to find out what people will prefer
| more satisfied with the program = receptive and
attentive to the information presented
| group of 30 employees randomly assigned to
each of the three conditions
| at the conclusion of the program (identical for all
3 groups) assess satisfaction with the program
|
12
4
The data
Condition Satisfaction Cond. Satis. Cond.
Satis.
days
5
nights
5 Saturday
5
days
5
nights
4 Saturday
5
days
5
nights
4 Saturday
5
days
5
nights
3 Saturday
4
days
4
nights
3 Saturday
4
days
4
nights
2 Saturday
4
days
4
nights
2 Saturday
3
days
4
nights
2 Saturday
3
days
3
nights
1 Saturday
3
days
2
nights
0 Saturday
2
13
Calculating F
1681 + 676 + 1444
10
2(df )
X days
5
5
5
5
1052
4
−
4
30 = 6.3
4
4
3
2
41
ΣX
2
1681
(ΣX)
Variance
0.99
Mean
4.1
Average variance
MSw
1.44
f=MSb/MSw=
fcrit(2,27) p