Two-Dimensional Probabilistic Risk Assessment Model 1523
Table VI. Top-Down Correlation Matrix for Input Rankings
with Different Sensitivity Analysis Methods Top-Down Correlation Results
b
Method
a
ANOVA PCA
SCA −
0.04 0.10
ANOVA −0.34, 0.36
−0.28, 0.45 PCA
0.59 0.26, 0.87 11
a
ANOVA: analysis of variance; PCA: Pearson correlation analysis; SCA: Spearman correlation analysis.
b
For each pair of methods, mean, 95 probability range, and number of times that the top-down correlation coefficient was
larger than 0.8 in uncertainty realizations are given.
r
The results based on Pearson and Spearman correlation typically were different compared
to those based on ANOVA with respect to identification of key important inputs. How-
ever, the two correlation-based methods iden- tified similar results with respect to unimpor-
tant inputs compared to ANOVA.
Table VII. Summary of the ANOVA
Results for Comingled Analysis of Variability and Uncertainty R
2
= 0.82
Equal F
Equal Rank Group
Group Rank
Rank Factor
a
Value Rank
in 2D
b
in 2D
c
in 2D
c
PCC
d
SCC
e
Temp1 1130
2 17
2 85
5 7
Temp2 0.1
13 7
4 91
13 12
Temp3 1035
3 25
2 97
1 4
Time1 550
4 31
2 86
6 1
Time2 0.3
12 7
4 100
12 13
Time3 5350
1 72
1 72
2 2
MD 34
9 11
4 92
9 13
LP1 261
5 38
3 66
7 6
LP2 0.7
11 10
4 95
10 11
LP3 200
6 32
3 73
4 3
GT1 52
7 16
4 84
8 8
GT2 1.0
10 17
4 92
11 10
GT3 45
8 12
4 85
3 5
Temp1 × Time1 1190
Temp2 × Time2 0.04
Temp3 × Time3 2270
a
The abbreviations used for factors in this table are the same as those defined in Table I.
b
Percent of times that rank in the comingled analysis equals the rank based on 2D analysis.
c
Group 1: most important input i.e., rank 1; Group 2: secondary importance inputs i.e., ranks between 2 and 4; Group 3: minor importance inputs i.e., ranks between 5 and 6;
Group 4: unimportant inputs i.e., ranks between 7 and 13.
d
Rank based on Pearson correlation analysis.
e
Rank based on Spearman correlation analysis.
4.3. Comingled Analysis of Variability and Uncertainty
4.3.1. Rankings based on ANOVA Table VII summarizes results when variability
and uncertainty were comingled into one dimension. The main effects of all factors and interaction effects
between storage temperature and storage time at all three stages were evaluated. For each factor the num-
ber of times that the comingled analysis provided the same ranking as the two-dimensional sensitivity
analysis was quantified.
Typically, the F values based on the comin- gled analysis of variability and uncertainty were sub-
stantially larger than those for the two-dimensional analysis. This is in large part because there was a sub-
stantially larger simulation sample size for the levels of each factor, which in turn reduces the sampling
error.
Except for the most important factor Time3, there was typically low concordance between the two
probabilistic approaches, where concordance refers to obtaining the same rank for a given input from
each of the two probabilistic approaches. For example,
1524 Mokhtari and Frey
Temp1 had a rank of 2 in the comingled analysis.
However, Temp1 had a rank of 2 in only 17 out of 100 uncertainty realizations of the model in the two-
dimensional analysis. In contrast, when grouping factors with compara-
ble importance, there was typically high concordance between the two probabilistic approaches. For exam-
ple, Temp1 was assigned to Group 2 based on the comingled analysis. Similarly, in 85 out of 100 uncer-
tainty realizations, Temp1 had a rank between 2 and 4, and hence was assigned to the same importance
group.
There was a statistically significant interaction be- tween storage time and storage temperature at Stages
1 and 3. The relative magnitude of F values indicated that the interaction effect between these two factors
at Stage 3 had higher importance. Similar to the two- dimensional analysis, there was no statistically signif-
icant interaction between storage time and storage temperature at Stage 2.
4.3.2. Rankings Based on Pearson and Spearman Correlation Analyses
Rankings based on Pearson and Spearman corre- lation analyses typically were different from ANOVA
results, except for inputs related to Stage 2. All three methods identified no statistically significant effects
for inputs related to Stage 2. Groups of inputs with comparable importance based on Pearson and Spear-
man correlation analysis results were different from those identified by ANOVA. For example, the most
important inputs based on Pearson and Spearman correlation analysis were Temp3 and Time1, respec-
tively, which were different from Time3 selected by ANOVA. Similarly, there was lack of agreement for
inputs selected in secondary and minor importance groups based on these methods.
4.3.3. Quantification of Sampling Distribution of F Values
When performing one-dimensional probabilistic analysis, a key question is regarding how large the
differences in F values between two factors must be in order to clearly discriminate their importance. A
method for answering this question is illustrated. The range of sampling distributions of F values provides
insight regarding how large the ratio of two F values must be in order for the ranks of the corresponding
factors to be substantially different. Empirical bootstrap simulation was used to gen-
erate sampling distributions of uncertainty for F values.
44
In the empirical bootstrap approach, an al- ternative randomized version of the original Monte
Carlo simulation was obtained by sampling with replacement from the original set of random values.
ANOVA was applied to each of the bootstrap samples to produce a distribution of F values. Summary of the
results from 200 bootstrap simulations are given in Table VIII. The summary table is comprised of mean
F
values, 95 probability range of F values, and coef- ficients of variation. The percentage of the bootstrap
simulations that produced a statistically significant F value with a significance level of 5 for each factor is
quantified. The arithmetic mean of 200 ranks and the range of ranks for each factor also are quantified.
There was a substantial range of uncertainty as- sociated with the estimates of F value. For example,
Temp1 was estimated to have a mean rank of 2.9. The
F value for this factor had a mean of 1445 and 95
probability range of 873–2040, or approximately plus or minus 40 of the mean. Temp3 had a mean rank
of 3.7, a mean F value of 1130, and a 95probability range of 650–1670, or approximately plus or minus
48. The sampling distributions of the F values for these two factors did not have statistically significant
correlation. Thus, the overlap in the confidence in- tervals of Temp1 and Temp3 indicated a possibility
that the rank order between these two factors could reverse, even though on average the F value for Time1
was larger than that for Temp3 by a factor of 1.3. This reversal in ranks happened in 32 of bootstrap sim-
ulations. In contrast, Temp3 had a statistically signif- icantly larger F value than the factor with the fifth
highest average rank, which was LP3. The 95 prob- ability ranges for F values of these two factors did not
overlap. Therefore, although there was some ambigu- ity regarding which of the factors might be the third
most important, it was clear that Temp1, and Temp3 were more important than LP3.
The coefficient of variation was approximately 0.30 or less for F values greater than approximately
200. For F values smaller than approximately 20, the coefficient of variation ranged from approximately 0.3
to 1.85. These results suggested that the coefficient of variation might be relatively constant for F values that
were statistically significant and substantially large. Since a 95 probability range might typically be en-
closed by plus or minus two standard deviations for a symmetric sampling distribution, one might infer that
differences in F values of approximately 60 or more imply a clear discrimination in rank order. This value
Two-Dimensional Probabilistic Risk Assessment Model 1525
Table VIII. Summary of the ANOVA Results for 200 Bootstrap Simulations
for F Value Statistics 95
Range Mean
Probability Frequency
c
Mean of
Factor
a
F Value
Range SD
Mean
b
Rank Rank
Temp1 1445
873, 2040 0.21
100 2.9
2–4 Temp2
2.7 0.3, 7.8
0.83 33
10.9 10–13
Temp3 1130
650, 1670 0.32
100 3.7
2–4 Time1
1550 1275, 1820
0.10 100
2.5 2–4
Time2 0.7
0.0, 3.6 1.75
2 12.5
10–13 Time3
6060 5250, 6860
0.05 100
1.0 1
d
MD 29
14, 45 0.41
100 8.9
8–9 LP1
273 236, 315
0.07 100
5.8 5–6
LP2 1.7
0.3, 4.3 0.68
22 11.4
10–13 LP3
325 295, 379
0.07 100
5.0 5–6
GT1 75
52, 97 0.26
100 7.0
7–8 GT2
1.9 0.3, 5.3
0.70 25
11.3 10–13
GT3 43
27, 55 0.29
100 8.1
7–9 Temp1 × Time1
1920 1660, 2150
0.07 100
Temp2 × Time2 0.7
0.3, 3.5 1.85
2 Temp3 × Time3
3605 3270, 4060
0.06 100
a
The abbreviations used for factors in this table are the same as those defined in Table I.
b
The ratio of standard deviation SD to mean F values for each factor also referred to as coefficient of variation.
c
The percentage of the bootstrap simulations for which the F values were statistically significant.
d
Time3 consistently had rank 1 in bootstrap simulations.
was specific to the example provided here and should not be used to make general quantitative judgments
regarding differences between F values obtained with different sample sizes or models. However, the tech-
nique used here can be applied to other case studies. The case study results suggest that the model output
has similar sensitivity to two or more factors if they have similar F values.
4.3.4. Summary of the Results for the Comingled Analysis of Variability and Uncertainty
Key insights and findings based on the comingled analysis of variability and uncertainty include:
r
Comingled analysis typically provided differ- ent results with respect to rank of impor-
tant inputs compared to the two-dimensional analysis. However, both probabilistic ap-
proaches identified similar groups of inputs with comparable importance.
r
Similar to the two-dimensional case, the two correlation-based methods failed to provide
the same insights with respect to identifica- tion of the most important inputs compared
to ANOVA.
r
In order to have statistically significantly differ- ent ranks, the two factors should have F values
that differ at least by 60 for the conditions of the case study.
4.4. Identifying Special Model Characteristics Using ANOVA
