05a Testing Model Fit 2005
Testing Model Fit
SOC 681
James G. Anderson, PhD
Limitations of Fit Indices
• Values of fit indices indicate only the average or
overall fit of a model. It is thus possible that some
parts of the model may poorly fit the data.
• Becauwe a single index reflects only a particul;ar
aspect oof model fit, a favorable value of that
indesx does not by itself indicate good fit. That is
why model fit is assessed based on the values of
multiple indices.
Limitations of Fit Statistics
• Fit indices do not indicate whether the
results are theoretically meaningful.
• Values of fit indices that suggest adequate
fit do not indicate the predictive power of
the model is also high.
• The sampling distribution of many fit
indices are unknown.
Assessment of Model Fit
• Examine the parameter estimates
• Examine the standard errors and significance
of the parameter estimates.
• Examine the squared multiple correlation
coefficients for the equations
• Examine the fit statistics
• Examine the standardized residuals
• Examine the modification indices
Measures of Fit
• Measures of fit are provided for three models:
– Default Model – this is the model that you specified
– Saturated Model – This is the most general model
possible. No constraints are placed on the population
moments It is guaranteed to fit any set of data perfectly.
– Independence Model – The observed variables are
assumed to be uncorrelated with one another.
Default Model
GPA
ACADEMIC
1
e1
HEIGHT
WEIGHT
RATING
ATTRACT
1
e2
Saturated Model
GPA
ACADEMIC
1
e1
HEIGHT
WEIGHT
RATING
ATTRACT
1
e2
Independence Model
GPA
ACADEMIC
HEIGHT
WEIGHT
RATING
ATTRACT
Overall measures of Fit
• NPAR is the number of parameters being
estimated (q)
• CMIN is the minimum value of the discrepancy
function between the sample covariance matrix
and the estimated covariance matrix.
• DF is the number of degrees of freedom and
equals the p-q
– p=the number of sample moments
– q= the number of parameters estimated
Overall measures of Fit
• CMIN is distributed as chi square with
df=p-q
• P is the probability of getting as large a
discrepancy with the present sample
• CMIN/DF is the ratio of the minimum
discrepancy to degrees of freedom. Values
should be close to 1.0 for correct models.
Chi Square: 2
• Best for models with N=75 to N=100
• For N>100, chi square is almost always
significant since the magnitude is
affected by the sample size
• Chi square is also affected by the size
of correlations in the model: the larger
the correlations, the poorer the fit
Chi Square to df Ratio: 2/df
• There are no consistent standards for
what is considered an acceptable
model
• Some authors suggest a ratio of 2 to 1
• In general, a lower chi square to df ratio
indicates a better fitting model
Transforming Chi Square to Z
Z = (2*2) - (2*df-1)
CMIN
Model
NPAR
Default model
22
Saturated model
36
Independence
model
CMIN DF
P CMIN/DF
10.335 14 .737
.000
.738
0
8 243.768 28 .000
8.706
RMR, GFI
• RMR is the Root Mean Square Residual. It
is the square root of the average amount
that the sample variances and covariances
differ from their estimates. Smaller values
are better
• GFI is the Goodness of Fit Index. GFI is
between 0 and 1 where 1 indicates a perfect
fit. Acceptable values are above 0.90.
GFI and AGFI
(LISREL measures)
• The AGFI takes into consideration the df
available to test the model.
• Values close to .90 reflect a good fit.
• These indices are affected by sample size
and can be large for poorly specified models.
• These are usually not the best measures to
use.
RMR, GFI
• AGFI is the Adjusted Goodness of Fit
Index. It takes into account the degrees of
freedom available for testing the model.
Acceptable values are above 0.90.
• PGFI is the Parsimony Goodness of Fit
Index. It takes into account the degrees of
freedom available for testing the model.
Acceptable values are above 0.90.
RMR, GFI
Model
RMR
GFI AGFI PGFI
Default model
.003
.975
Saturated model
.000 1.000
Independence
model
.023
.570
.935
.379
.447
.443
Comparisons to a Baseline Model
• NFI is the Normed Fit Index. It compares
the improvement in the minimum
discrepancy for the specified (default)
model to the discrepancy for the
Independence model. A value of the NFI
below 0.90 indicates that the model can be
improved.
Bentler-Bonett Index or
Normed Fit Index (NFI)
• Define null model in which all correlations
are zero:
2 (Null Model) - 2 (Proposed Model)
(Null Model)
2
• Value between .90 and .95 is acceptable;
above .95 is good
• A disadvantage of this index is that the
more parameters, the larger the index.
Comparisons to a Baseline Model
• RFI is the Relative Fit Index This index takes the
degrees of freedom for the two models into account.
• IFI is the incremental fit index. Values close to 1.0
indicate a good fit.
• TLI is the Tucker-Lewis Coefficient and also is
known as the Bentler-Bonett non-normed fit index
(NNFI). Values close to 1.0 indicate a good fit.
• CFI is the Comparative Fit Index and also the
Relative Noncentrality Iindex (RNI). Values close to
1.0 indicate a good fit.
Tucker Lewis Index or Nonnormed Fit Index (NNFI)
• Value: 2/df(Null Model) - 2/df(Proposed
Model)
2/df(Null Model)
• If the index is greater than one, it is set to1.
• Values close to .90 reflects a good model fit.
• For a given model, a lower chi-square to df
ratio (as long as it is not less than one) implies
a better fit.
Comparative Fit Index (CFI)
• If D= 2 - df, then:
•
•
•
•
D(Null Model) - D(Proposed Model)
D(Null Model)
If index > 1, it is set to 1; if index .10 indicate a poor fit.
• It is a parsimony-adjusted measure.
• Amos provides upper and lower limits of a
90% confidence interval for the RMSEA
PCLOSE
• PCLOSE is the probability for testing the
null hypothesis that the population RMSEA
is no greater than 0.05.
RMSEA
Model
RMSEA LO 90 HI 90 PCLOSE
Default model
.000
.000
.072
.877
Independence
model
.282
.250
.315
.000
Information Theoretic Measures
• These indices are composite measures of
badness of fit and complexity.
• Simple models that fit well receive low
scores. Complicated poorly fitting models
get high scores.
• These indices are used for model
comparison not to evaluate a single model.
AIC
Model
AIC
BCC
BIC
CAIC
Default model
54.335
58.835 111.204 133.204
Saturated model
72.000
79.364 165.059 201.059
Independence
model
259.768 261.404 280.447 288.447
Akaike Information Criterion
(AIC)
• Value: 2 + k(k-1) - 2(df)
where k= number of variables in the model
• A better fit is indicated when AIC is smaller
• Not standardized and not interpreted for a
given model.
• For two models estimated from the same data,
the model with the smaller AIC is preferred.
Information Theoretic Measures
•
•
•
•
BCC is the Browne-Cudeck Criterion
BIC is Bayes Information Criterion.
CAIC is the consistent AIC
ECVI except for a constant scale factor it is
the same as AIC.
• MECVI except for a scale factor is the same
as the BCC.
ECVI
Model
ECVI LO 90 HI 90 MECVI
Default model
.560
.598
.671
.607
Saturated model
.742
.742
.742
.818
2.678 2.202 3.231
2.695
Independence
model
Nonhierarchical Models
Illness
Symptoms
Diminished SES
Low Morale
Neurological
Dysfunction
Poor
Relationships
Illness
Symptoms
Diminished SES
Neurological
Dysfunction
Low
Morale
Poor
Relationships
Hierarchical Models
Difference in Chi Square
Value: X2diff = X2 model 1 -X2 model 2
DFdiff = DF model 1 –DFmodel 2
Illness
Symptoms
Diminished SES
Low Morale
Neurological
Dysfunction
Poor
Relationships
Illness
Symptoms
Diminished SES
Low Morale
Neurological
Dysfunction
Poor
Relationships
Miscellaneous Measures
• HOELTER is the largest sample size for
which one would accept the hypothesis that
a model is correct.
Hoelter Index
• Value: (N-1)*2(crit) + 1
2
Where 2 (crit) is the critical value for the
chi-square statistic
• The index should only be calculated if
chi square is statistically significant.
Hoelter Index (2)
• If the critical value is unknown, can
approximate: [ (1.645 + (2df-1) ]2 + 1
2 2/ (N-1)
• For both formulas, one rounds down to
the nearest integer
• The index states the sample size at
which the chi square would not be
significant
Hoelter Index (3)
• In other words, how small one’s sample
size would have to be for chi square to
no longer be significant
• Hoelter Recommends values of at least
200
• Values < 75 indicate poor fit
HOELTER
Model
HOELTER HOELTER
.05
.01
Default model
223
274
Independence
model
17
20
Which Fit Indices to Report?/
•
•
•
•
•
Chi Square and df
RMSEA
CFI
AGFI
Hierarchical Models: Difference in Chi
Square
• Nonhierarchical Models: AIC
SOC 681
James G. Anderson, PhD
Limitations of Fit Indices
• Values of fit indices indicate only the average or
overall fit of a model. It is thus possible that some
parts of the model may poorly fit the data.
• Becauwe a single index reflects only a particul;ar
aspect oof model fit, a favorable value of that
indesx does not by itself indicate good fit. That is
why model fit is assessed based on the values of
multiple indices.
Limitations of Fit Statistics
• Fit indices do not indicate whether the
results are theoretically meaningful.
• Values of fit indices that suggest adequate
fit do not indicate the predictive power of
the model is also high.
• The sampling distribution of many fit
indices are unknown.
Assessment of Model Fit
• Examine the parameter estimates
• Examine the standard errors and significance
of the parameter estimates.
• Examine the squared multiple correlation
coefficients for the equations
• Examine the fit statistics
• Examine the standardized residuals
• Examine the modification indices
Measures of Fit
• Measures of fit are provided for three models:
– Default Model – this is the model that you specified
– Saturated Model – This is the most general model
possible. No constraints are placed on the population
moments It is guaranteed to fit any set of data perfectly.
– Independence Model – The observed variables are
assumed to be uncorrelated with one another.
Default Model
GPA
ACADEMIC
1
e1
HEIGHT
WEIGHT
RATING
ATTRACT
1
e2
Saturated Model
GPA
ACADEMIC
1
e1
HEIGHT
WEIGHT
RATING
ATTRACT
1
e2
Independence Model
GPA
ACADEMIC
HEIGHT
WEIGHT
RATING
ATTRACT
Overall measures of Fit
• NPAR is the number of parameters being
estimated (q)
• CMIN is the minimum value of the discrepancy
function between the sample covariance matrix
and the estimated covariance matrix.
• DF is the number of degrees of freedom and
equals the p-q
– p=the number of sample moments
– q= the number of parameters estimated
Overall measures of Fit
• CMIN is distributed as chi square with
df=p-q
• P is the probability of getting as large a
discrepancy with the present sample
• CMIN/DF is the ratio of the minimum
discrepancy to degrees of freedom. Values
should be close to 1.0 for correct models.
Chi Square: 2
• Best for models with N=75 to N=100
• For N>100, chi square is almost always
significant since the magnitude is
affected by the sample size
• Chi square is also affected by the size
of correlations in the model: the larger
the correlations, the poorer the fit
Chi Square to df Ratio: 2/df
• There are no consistent standards for
what is considered an acceptable
model
• Some authors suggest a ratio of 2 to 1
• In general, a lower chi square to df ratio
indicates a better fitting model
Transforming Chi Square to Z
Z = (2*2) - (2*df-1)
CMIN
Model
NPAR
Default model
22
Saturated model
36
Independence
model
CMIN DF
P CMIN/DF
10.335 14 .737
.000
.738
0
8 243.768 28 .000
8.706
RMR, GFI
• RMR is the Root Mean Square Residual. It
is the square root of the average amount
that the sample variances and covariances
differ from their estimates. Smaller values
are better
• GFI is the Goodness of Fit Index. GFI is
between 0 and 1 where 1 indicates a perfect
fit. Acceptable values are above 0.90.
GFI and AGFI
(LISREL measures)
• The AGFI takes into consideration the df
available to test the model.
• Values close to .90 reflect a good fit.
• These indices are affected by sample size
and can be large for poorly specified models.
• These are usually not the best measures to
use.
RMR, GFI
• AGFI is the Adjusted Goodness of Fit
Index. It takes into account the degrees of
freedom available for testing the model.
Acceptable values are above 0.90.
• PGFI is the Parsimony Goodness of Fit
Index. It takes into account the degrees of
freedom available for testing the model.
Acceptable values are above 0.90.
RMR, GFI
Model
RMR
GFI AGFI PGFI
Default model
.003
.975
Saturated model
.000 1.000
Independence
model
.023
.570
.935
.379
.447
.443
Comparisons to a Baseline Model
• NFI is the Normed Fit Index. It compares
the improvement in the minimum
discrepancy for the specified (default)
model to the discrepancy for the
Independence model. A value of the NFI
below 0.90 indicates that the model can be
improved.
Bentler-Bonett Index or
Normed Fit Index (NFI)
• Define null model in which all correlations
are zero:
2 (Null Model) - 2 (Proposed Model)
(Null Model)
2
• Value between .90 and .95 is acceptable;
above .95 is good
• A disadvantage of this index is that the
more parameters, the larger the index.
Comparisons to a Baseline Model
• RFI is the Relative Fit Index This index takes the
degrees of freedom for the two models into account.
• IFI is the incremental fit index. Values close to 1.0
indicate a good fit.
• TLI is the Tucker-Lewis Coefficient and also is
known as the Bentler-Bonett non-normed fit index
(NNFI). Values close to 1.0 indicate a good fit.
• CFI is the Comparative Fit Index and also the
Relative Noncentrality Iindex (RNI). Values close to
1.0 indicate a good fit.
Tucker Lewis Index or Nonnormed Fit Index (NNFI)
• Value: 2/df(Null Model) - 2/df(Proposed
Model)
2/df(Null Model)
• If the index is greater than one, it is set to1.
• Values close to .90 reflects a good model fit.
• For a given model, a lower chi-square to df
ratio (as long as it is not less than one) implies
a better fit.
Comparative Fit Index (CFI)
• If D= 2 - df, then:
•
•
•
•
D(Null Model) - D(Proposed Model)
D(Null Model)
If index > 1, it is set to 1; if index .10 indicate a poor fit.
• It is a parsimony-adjusted measure.
• Amos provides upper and lower limits of a
90% confidence interval for the RMSEA
PCLOSE
• PCLOSE is the probability for testing the
null hypothesis that the population RMSEA
is no greater than 0.05.
RMSEA
Model
RMSEA LO 90 HI 90 PCLOSE
Default model
.000
.000
.072
.877
Independence
model
.282
.250
.315
.000
Information Theoretic Measures
• These indices are composite measures of
badness of fit and complexity.
• Simple models that fit well receive low
scores. Complicated poorly fitting models
get high scores.
• These indices are used for model
comparison not to evaluate a single model.
AIC
Model
AIC
BCC
BIC
CAIC
Default model
54.335
58.835 111.204 133.204
Saturated model
72.000
79.364 165.059 201.059
Independence
model
259.768 261.404 280.447 288.447
Akaike Information Criterion
(AIC)
• Value: 2 + k(k-1) - 2(df)
where k= number of variables in the model
• A better fit is indicated when AIC is smaller
• Not standardized and not interpreted for a
given model.
• For two models estimated from the same data,
the model with the smaller AIC is preferred.
Information Theoretic Measures
•
•
•
•
BCC is the Browne-Cudeck Criterion
BIC is Bayes Information Criterion.
CAIC is the consistent AIC
ECVI except for a constant scale factor it is
the same as AIC.
• MECVI except for a scale factor is the same
as the BCC.
ECVI
Model
ECVI LO 90 HI 90 MECVI
Default model
.560
.598
.671
.607
Saturated model
.742
.742
.742
.818
2.678 2.202 3.231
2.695
Independence
model
Nonhierarchical Models
Illness
Symptoms
Diminished SES
Low Morale
Neurological
Dysfunction
Poor
Relationships
Illness
Symptoms
Diminished SES
Neurological
Dysfunction
Low
Morale
Poor
Relationships
Hierarchical Models
Difference in Chi Square
Value: X2diff = X2 model 1 -X2 model 2
DFdiff = DF model 1 –DFmodel 2
Illness
Symptoms
Diminished SES
Low Morale
Neurological
Dysfunction
Poor
Relationships
Illness
Symptoms
Diminished SES
Low Morale
Neurological
Dysfunction
Poor
Relationships
Miscellaneous Measures
• HOELTER is the largest sample size for
which one would accept the hypothesis that
a model is correct.
Hoelter Index
• Value: (N-1)*2(crit) + 1
2
Where 2 (crit) is the critical value for the
chi-square statistic
• The index should only be calculated if
chi square is statistically significant.
Hoelter Index (2)
• If the critical value is unknown, can
approximate: [ (1.645 + (2df-1) ]2 + 1
2 2/ (N-1)
• For both formulas, one rounds down to
the nearest integer
• The index states the sample size at
which the chi square would not be
significant
Hoelter Index (3)
• In other words, how small one’s sample
size would have to be for chi square to
no longer be significant
• Hoelter Recommends values of at least
200
• Values < 75 indicate poor fit
HOELTER
Model
HOELTER HOELTER
.05
.01
Default model
223
274
Independence
model
17
20
Which Fit Indices to Report?/
•
•
•
•
•
Chi Square and df
RMSEA
CFI
AGFI
Hierarchical Models: Difference in Chi
Square
• Nonhierarchical Models: AIC