CONFIDENCE INTERVAL ESTIMATION
9.4 CONFIDENCE INTERVAL ESTIMATION
Confidence interval estimation quantifies the confidence (probability) that the true (but unknown) statistical parameter falls within an interval whose boundaries are calculated using appropriate point estimates (see Section 3.10). Standard statistical procedures for confidence interval estimation assume that the underlying sample consists of iid observations. Recall from Section 9.2.1 that one way of obtaining iid observations is to generate multiple replications whose random number streams are independent. Recall further from Section 9.2.2 that in a steady-state simulation setting, one can also use the batch means method to obtain multiple estimates from a single replication. For more information, see Alexopoulos and Seila (1998) and Nelson (1992).
9.4.1 C ONFIDENCE I NTERVALS FOR T ERMINATING S IMULATIONS
In this section we illustrate confidence interval estimation of an (unknown) param- eter y in a terminating simulation. Following the setting in Section 9.1, we assume that n independent fixed-length replications of the model were run, and produced a sample
f ^ y (1), . . . , ^ y (n) g, where ^ y (r) is the point estimate of y produced by replication r. The pooled point estimator for y, is the sample mean across replications
Y ¼ ^ (r)
n r ¼1
174 Output Analysis and the corresponding pooled estimate y is obtained from Eq. 9.6 by substituting
Y ^ (r) y ^ ¼ (r) on its right-hand side for each r ¼ 1, . . . , n. Note carefully that the Y in Eq. 9.6 is a random variable with mean m and variance s 2 =n. Thus, Y , and consequently, increase our confidence in the point estimate value y
Y . However, we aim to further quantify this confidence by computing (at least approxi- mately) the probability of events of the form
Pr ^ f Y 1 Y ^ 2 g¼1 a (9 :7) where the estimators ^ Y 1 and ^ Y 2 define a (random) confidence interval ^ ½ Y 1 ,^ Y 2 for y,
and a is the probability that the confidence interval does not include y (a is a small probability, typically around 0.05). Recall from Section 3.10 that Eq. 9.7 specifies a confidence interval at confidence level (or significance level) a. To be able to obtain confidence interval estimates from Eq. 9.7, two conditions should hold:
Y is known, at least approximately. Y can be expressed in terms of the unknown parameter y.
To satisfy condition 1, observe that the sample ^ f y (1), . . . , ^ y (n) g was drawn from
a set of iid random variables, f Y ^ (1), . . . , ^ Y (n) g, with common mean m and common variance s 2 . Under the central limit theorem (see Section 3.8.5), the random variable P n ^
r ¼1 Y (r) in Eq. 9.6 is approximately normally distributed with mean nm, and variance ns 2 . From properties of the normal distribution, it follows that
2 =n) (approximately), and the approximation improves as the sample size, n, tends to infinity.
To satisfy condition 2 we shall assume that the estimators ^ Y (r) are unbiased for y, so that we can write
(9 :8) (or at least approximately so, if the bias is tolerable).
m ¼ E½ Y ¼ E½ Y ^ (r) ¼y
In practice, it is convenient to express confidence interval probabilities of the form in Y , but also in terms of the standardized normal random variable
s = ffiffiffi n
a g ¼ a. The reason is that tables for z a as function of a are widely available in books and in computer programs, and there is no need to have separate tables for other values of m and s 2 . To see how this works, see Figure 9.2. Note that z a =2 ¼z 1a =2 and the probability is written as follows:
and its quantiles z a , where Pr
1a =2 g¼1 a (9 :10) Next, substitute Z from Eq. 9.9 into Eq. 9.10. Since m ¼ y, simple algebra then yields
Pr fz 1a =2
the final representation of a confidence interval for the mean value at level a as p ffiffiffiffiffiffiffiffiffiffi 2 p Pr ffiffiffiffiffiffiffiffiffiffi f Y z
1a =2
s =n
Y þz 1a =2 s 2 =n g¼1 a ; (9 :11)
Output Analysis 175
z 1-a/2
Figure 9.2 Confidence interval estimation using the standardized normal distribution.
which is often abbreviated as
(9 :12) The notation in Eq. 9.12 highlights the fact that the confidence interval for the mean p ffiffiffiffiffiffiffiffiffiffi
p Y ffiffiffiffiffiffiffiffiffiffi
1a =2
s 2 =n :
Y , with half-width z 1a =2 2 s =n . Note that for a fixed a, the half-width is a measure of the accuracy of the associated confidence interval estimator: The narrower it is, the smaller its variance. But since the quantile z 1a =2 2 and the variance s are fixed values, it is clear from Eq. 9.11 that the accuracy of a confidence interval can be enhanced only by increasing the number of replications, n. This analysis conforms to the old adage of “no free lunch”: To increase the statistical accuracy of estimates, one must pay in the coin of additional computation.
Finally, the confidence interval in Eq. 9.11 assumes that the variance s 2 is known. Unfortunately, this is often not the case, and s 2 needs to be estimated from the sample f^ Y (1), . . . , ^ Y (n) g by the sample variance (see Section 3.10),
½^ Y (r) Y 2 :
n 1 r ¼1
Substituting the sample variance above into Eq. (9.9) results in a new random variable
distributed according to the Student t distribution with n – 1 degrees of freedom (see Section 3.8.8). This distribution has a slightly larger variance than the corresponding
standard normal distribution, Norm(0,1), but it converges to it as n tends to infinity. The quantile of the Student t distribution with n degrees of freedom at significance level a is denoted by t n, a , namely, Pr fT n
g ¼ a. The confidence interval in this case becomes q Y ffiffiffiffiffiffiffiffiffiffiffi
n, a
(9 :12) For a given significance level a, the increased variance of the Student t distribution
n 1, 1 a
S =2 2 Y ^ =n :
relative to the standard normal distribution will in turn tend to result in confidence intervals in Eq. 9.14 that are slightly wider than their normal counterparts in Eq. 9.12.
176 Output Analysis This is due to the added randomness in estimating the underlying variance by the
estimator S 2 Y ^ . For a detailed discussion of output analysis in terminating simulations, refer to Law (1980) and Law and Kelton (2000).
9.4.2 C ONFIDENCE I NTERVALS FOR S TEADY -S TATE S IMULATIONS
The methodology for estimating confidence intervals from replications of a termin- ating simulation (see Section 9.4.1) carries over to the present case of steady-state simulation. However, steady-state replications tend to be longer than their terminating counterparts, and consequently take longer to compute. On the other hand, longer replications often allow the application of the batch means method.
Consider next confidence interval estimation for some mean y ¼ m in a batch means setting with a single replication consisting of m batches (regardless of whether the underlying history was discrete or continuous). The corresponding set of m estimators,
f Y ^ 1 ,...,^ Y m g, gives rise to a sample mean
m j ¼1
and sample variance
Y j Y : m 1 j ½ ¼1
Finally, the confidence interval for y ¼ m at significance level a is
q Y ffiffiffiffiffiffiffiffiffiffiffiffi
m 1, 1 a =2
S 2 Y ^ =m :
For other approaches to the construction of confidence intervals, including methods producing a desired half-width, refer to Banks et al. (2004), Brately et al. (1987), Fishman and Yarberry (1997), Law (1977), Law and Kelton (1982), and Law and Kelton (2000).
9.4.3 C ONFIDENCE I NTERVAL E STIMATION IN A RENA
Standard Arena output (see Sections 5.4 and 5.5) provides 95% batch means confi- dence intervals for each replication. These confidence intervals are computed for both Tally and Time Persistent statistics in terms of half-widths under the Half Width column heading. If, however, the estimated batch means are significantly dependent or the underlying sample history is too short to yield a sufficient number of batches, that column will display the message (Insufficient) to indicate that the data are not appropriate or inadequate for confidence interval estimation.
Arena also supports the computation of confidence intervals from multiple replica- tions as a SIMAN summary report. The analyst can request this report by selecting the Setup. . . option in the Run menu, clicking on its Reports tab, and finally selecting the option SIMAN Summary Report(.out file) in the Default Report field. For Arena model name some model, the report is placed in file some model :out, and can be examined in any text editor.
Output Analysis 177