Directory UMM :Data Elmu:jurnal:O:Operations Research Letters:Vol26.Issue2.Mar2000:
Operations Research Letters 26 (2000) 81–89
www.elsevier.com/locate/orms
Ecient data allocation for frequency domain experiments (
Susan M. Sanchez a; ∗ , Prabhudev Konanab
a Management
Science and Information Systems, University of Missouri-St. Louis, School of Business Administration,
St. Louis, MO 63121-4499, USA
b Department of Management Science and Information Systems, The University of Texas at Austin, USA
Received 1 July 1995; received in revised form 1 September 1999
Abstract
Frequency domain experiments are ecient screening procedures for identifying important factors in simulation models.
An empirical investigation involving a simple autoregressive system and a complex queueing system shows that allocating
data unequally among the signal and noise runs may be more e ective when the total data collection e ort is limited.
c 2000 Elsevier Science B.V. All rights reserved.
Keywords: Simulation; Design of experiments; Factor screening
1. Introduction
One of the goals of many simulation studies is to understand the relationships among various input factors
and how their levels a ect the system response. The
simplicity of factorial designs makes them attractive
to analysts interested in examining systems characterized by a moderately small number of factors or simple (linear) response relationships. However, despite
the recent improvements in computing power and
cost, the time required to conduct a thorough factorial
analysis of a simulated system may be prohibitive if
the number of input factors is large. If output data are
highly autocorrelated, then large portions may need to
( Supported in part by the University of Missouri-St. Louis
Oce of Research, and Grant DDM-9396135 from the National
Science Foundation.
∗ Corresponding author.
be truncated from each run in order to remove initialization bias e ects. Further, non-linear model structures with non-constant variance may also be needed
to adequately model a complex system’s behavior
and arrive at statistically valid assessments of the
system.
Schruben and Cogliano [14] proposed frequency
domain experimentation (FDE) as an alternative to traditional factor screening techniques in computer simulation (see also the work of Sanchez and Buss [9]).
FDE provides a powerful and convenient method for
simulation factor screening with just two runs: a noise
run and a signal run. The noise run acts as a control for any natural cyclic behavior of the system and
is generated by holding all factors at nominal levels.
The signal run is generated by sinusoidally oscillating
the factor levels, each at a di erent driving frequency,
f. These driving frequencies are chosen such that
the output terms of polynomial order correspond to
c 2000 Elsevier Science B.V. All rights reserved.
0167-6377/00/$ - see front matter
PII: S 0 1 6 7 - 6 3 7 7 ( 9 9 ) 0 0 0 7 0 - X
82
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
identi able indicator frequencies on the spectrum. For
binary factors, the probability of achieving a speci ed
level is oscillated sinusoidally. This approach extends
to qualitative factors with three or more levels via a
binary tree structure; levels for quantitative discrete
factors can also be approximated by continuous sine
functions [10].
Once the signal and noise runs have been generated,
signal and noise spectra can be computed using (fast)
Fourier transformations. The existence of important
factor e ects can then be assessed by examining the
spikes in the signal distribution at the corresponding
indicator frequencies. There are two approaches to
analyzing these spikes: the spectral ratio approach
[14] and the spectral di erence approach. In the
spectral ratio approach a factor is said to a ect the response if the ratio of the signal spectrum to the noise
spectrum at its identifying frequency is substantially
greater than one. The spectral ratio asymptotically
follows an F distribution. In the spectral di erence
approach [7,12], the noise spectral terms are subtracted from the signal spectral terms (by frequency)
and the resulting spectral di erences are asymptotically distributed 2 . In both situations, a qualitative
rather than quantitative approach is recommended
since the degrees of freedom associated with the F
or 2 distributions must be approximated when the
noise terms are correlated. (For a detailed explanation of FDE, along with code necessary to select the
driving frequencies and perform the Fourier transformations, see the on-line or o -line version of Sanchez
et al. [11].)
In past studies, data were allocated equally between
the noise and signal runs, more for convenience than
for any scienti c reason. How do allocation strategies
a ect the spectral tests? Will increasing the proportion
of data to the signal (or noise) run result in a greater
screening ability? Is there a desirable (optimal) absolute data amount for signal and noise runs? These
questions are of interest when experiments must be
conducted under xed budget or time constraints. In
this paper, we analyze the power of spectral ratio and
di erence tests under various data allocation strategies. Sections 2 and 3 contain empirical results from
a simple autoregressive system and 11-factor queueing system, respectively. We identify pitfalls, propose
preliminary guidelines for data allocation, and discuss
future research directions in Section 4.
2. Autoregressive system example
Consider the following single-factor linear model:
p
Yt = Xt + et ; et = 1 − 2 Zt + et−1
where Yt ; Xt and et are the response from the simulation model, the input factor level, and the noise term,
respectively, at time t. The Zt ’s are i.i.d. standard normal random variates. Xt is held at a nominal level of
zero during the noise run. During the signal run, Xt is
oscillated at Xt = m cos(2ft), where f is the driving
frequency in cycles per observation, and m is the amplitude of the oscillation. The et ’s are independent of
the input factor Xt , but are positively (or negatively)
autocorrelated depending on the value of the correlation coecient . To facilitate comparisons, the scaling factor is included so that the error variance is equal
to one for all ; thus m is the ratio of the factor e ect
to the error standard deviation.
2.1. Experimental design
For this simple linear system, there are ve factors
that might a ect the ability of the FDE to identify the
presence of the Xt term in the model: (1) the amplitude
of the input signal oscillation, (2) the autocorrelation
of the error distribution, (3) the total sample size, (4)
the driving frequency for the input signal, and (5) the
proportion of the total sample allocated to the signal
run. Of these, the rst two factors are characteristics
of the system itself. The oscillation amplitude plays
an important role in the factor screening process [4],
while the error correlation structure impacts the shape
of the noise spectrum. For the simple linear system we
study, the noise spectrum is
at if = 0, has power
concentrated at low frequencies (skewed right) for
¿ 0 and has power concentrated at high frequencies
(skewed left) for ¡ 0.
The nal three factors are speci ed by the experimenter. Let N denote the total sample size and
Ns denote the sample size allocated to the signal
run. The driving frequency for our single input factor can be speci ed at any value between 0 and
0.5 cycles=observation, although for more complex
modeling situations involving many input factors the
driving frequencies cannot be chosen with complete
freedom ([2], see [11] for code). Spectral ratio tests
will be more powerful if the driving frequencies are
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
those for which the noise spectral values (i.e., system gain) are low [14], but guidelines for allocation
should ideally be robust to the speci cation of driving
frequency values. (Often the analyst may not know
the system gain behavior a priori, and it may not be
possible to oscillate all factors at frequencies that
correspond to low noise spectral ratios during investigations of complex systems involving many factors.)
Our primary questions of interest are the e ect of
Ns =N on factor screening capability, and whether or
not this varies with N . Our experimental settings for
the ve factors are:
• Signal amplitude (m): 0:1; 0:5
• Correlation coecient (): − 0:9 to +0:9 (by increment of 0.2)
• Proportion of data allocated to signal (Ns =N ): 0:1
to 0.9 (by increment of 0.2)
• Total sample size (N ): 1000; 5000 and 10 000
• Driving frequency (f, in cycles=observation):
0:1; 0:4
Each combination of the factor levels (600 in all)
is a design point. 1000 independently seeded replications of the FDE are generated for each design point,
with signal and noise runs independently seeded. This
same set of random number seeds is common to all
design points for increased eciency in comparisons.
Fourier transforms are evaluated at the frequencies
f = 0(0:01)0:5 using a Tukey window [1, p. 114].
2.2. Results
We rst illustrate the performance of the method in
terms of overall identi cation ability. Let Imax denote
the number of times (out of the 1000 replications)
that the driving frequency yielded the largest spectral
ratio (or di erence) among the 51 frequencies evaluated. Fig. 1 graphs the results of our experiment
when the driving frequency f = 0:1 and amplitude
m = 0:1; 0:5 (the shaded surface corresponds to 0.1
and the unshaded corresponds to 0.5) for both spectral
ratios and spectral di erences. Each subgraph shows
Imax as a function of the allocation proportion Ns =N
and the error correlation . From these graphs, we see
that spectral ratios have the greatest power (i.e., the
highest probability of correctly detecting the model
term) for far from zero, and spectral di erences have
83
the greatest power for near zero. As expected, the
power increases as the sample size N increases or as
the amplitude of the factor level increases. Graphs for
f = 0:4 (not shown) essentially re
ect the curves of
Fig. 1 around = 0.
The allocation proportion clearly a ects the power
of both spectral tests, although the results are most
readily apparent for the case m = 0:1. For the most
part, the screening power is the lowest at Ns =N = 0:1;
the power increases as Ns =N increases to 0.7. Any
further increase in Ns =N results in lower or equal
spectral power for the spectral ratio approach, while
the power increases slightly for some of the spectral
di erence experiments. Similar gures result when
the driving frequency is changed. Thus, it appears
that identi cation can be improved for small-sample
FDEs by allocating more than half of the data to the
signal run.
We also consider the performance on an individual
experiment basis by examining the number of identi cations as functions of the data allocation and spectral frequency for a subgroup of the design points:
N = 1000 or 5000; = 0:7 and f = 0:1. We use this
subgroup for illustration because, as Fig. 1 indicates,
neither spectral approach does a good job of correctly
identifying the indicator frequency across all allocation proportions and both values of m. When spectral
ratios are used, then for low N and low m the frequencies most likely to be associated with the highest
spectral ratio are 0.0 and 0.5: the indicator frequency
is not selected particularly more than any of the other
frequencies, although the corresponding Imax is lowest
for Ns =N = 0:1. As the sample size increases, the indicator frequency is much more likely to yield the largest
spike: only the frequency of 0.5 also stands out against
the background noise. For the larger value of m, the
indicator frequency is chosen most often. Neighboring frequencies occasionally yield larger spectral ratios for small Ns =N , due to a phenomenon known as
the smearing of the spectrum. A pattern of screening power that increases sharply and then decreases
slightly with Ns =N occurs for N = 1000; the procedure
correctly identi es the indicator frequency in all cases
for Ns =N ¿0:3 when N = 5000.
The individual performance curves for spectral
di erences di er markedly from those for spectral
ratios. When m is small, false positives over a band
of low frequencies dominate the indicator frequency
84
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
Fig. 1. Correct identi cations for autoregressive system, amplitude m = 0:1 (shaded) and m = 0:5 (unshaded).
spike. This occurs, regardless of the driving frequency, because the noise spectrum is skewed right
( ¿ 0). If ¡ 0, then the false positives are clustered around frequencies near 0.5 rather than those
near zero. However, consistent with Fig. 1, the identi cation power increases and then stabilizes as Ns =N
increases.
A more detailed look at the performance is provided
in Table 1, in which terms have been averaged over
runs with non-identical factors in order to reduce the
impact of inherent system characteristics. To construct
this table we begin, for every one of the 600 design
points, by computing the mean and standard deviation of the spectral ratios (or di erences) for each of
51 frequencies (0.0 to 0.5 by 0.01) over 1000 replications. The results for the indicator frequencies are
further averaged over and m to create entries of Table 1 for the spike means and standard deviations. All
spectral ratios (or di erences) for the 50 non-indicator
frequencies have been averaged over and m, and
reported as the background means and standard deviations in this table. The results provide a robust
view of the e ects of data allocation on the screening
capability.
85
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
Table 1
Average magnitudes of spectral terms for autoregressive system
Ns =N
Frequency
Indicator
0.1
Background
Indicator
0.3
Background
Indicator
0.5
Background
Indicator
0.7
Background
Indicator
0.9
Background
Statistic
Spectral ratios
N
Spectral di erences
N
1000
5000
10 000
1000
Mean
Std. dev.
Mean
Std. dev.
4.54
1.92
1.23
0.72
Mean
Std. dev.
Mean
Std. dev.
8.43
2.02
1.10
0.41
11.29
2.15
1.04
0.33
0.31
0.24
−0:00
0.20
0.69
0.21
0.00
0.13
0.97
0.20
−0:00
0.11
7.06
2.64
1.21
0.60
13.91
3.08
1.03
0.34
19.01
3.43
1.03
0.28
0.53
0.23
0.01
0.16
1.20
0.20
−0:00
0.11
1.70
0.19
−0:00
0.09
Mean
Std. dev.
Mean
Std. dev.
9.12
3.51
1.20
0.60
17.84
4.13
1.05
0.33
24.40
4.66
1.03
0.27
0.69
0.22
0.01
0.16
1.55
0.20
0.00
0.11
2.19
0.19
−0:00
0.09
Mean
Std. dev.
Mean
Std. dev.
11.00
5.05
1.22
0.66
21.16
5.55
1.07
0.36
28.88
6.17
1.05
0.29
0.82
0.23
0.01
0.16
1.83
0.20
0.00
0.11
2.59
0.19
0.00
0.09
Mean
Std. dev.
Mean
Std. dev.
14.67
18.33
1.42
2.90
25.17
8.99
1.12
0.47
33.77
9.68
1.09
0.37
0.93
0.25
0.02
0.20
2.08
0.21
0.00
0.13
2.94
0.20
0.00
0.11
Because the term associated with the indicator
frequency may be selected as important even if it
does not yield the largest spectral ratio (or di erence), Table 1 provides some information about how
powerful the screening procedure might be for an
unknown (simple) system. For example, the mean
spectral ratio associated with the indicator frequency
is 3.5 times as large as the mean background ratio
when N = 1000 and Ns =N = 0:1. When Ns =N increases to 0.9 for this value of N , the mean spectral
ratio for the indicator frequency is 10 times as large
as the mean background ratio. This improvement
(roughly tripling the average spike for the indicator
frequency) also holds as Ns =N increases from 0.1 to
0.9 for the other two values of N , even though the
tests with larger N are initially more powerful. For
N = 10 000, the mean spectral ratio for the indicator
is about 11 times as large as the mean background
ratio when Ns =N = 0:1, and about 33 times as large
5000
10 000
as the mean background ratio when Ns =N = 0:9. The
spectral di erences show similar behavior, although
the scale is di erent. The average spectral di erences for the non-indicator frequencies conform with
theory, and are not statistically distinguishable from
zero. Increasing the allocation from Ns =N = 0:1 to
Ns =N = 0:9 roughly triples the magnitude of the spike
associated with the indicator frequency. This shows
that the power of both the spectral ratio and di erence
approaches improves as the allocation ratio increases,
at least over the range of allocations and sample sizes
examined.
3. African tanker example
Our second example is the African tanker problem
[3]. This problem has also been studied by Pritsker [6],
Schriber [13], and Sanchez [8]. A brief description of
the system follows.
86
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
A port in Africa is used to load tankers with crude
oil for overwater shipment, and the port has facilities for loading more than one tanker simultaneously. The tankers are of three di erent types and
have di erent service time distributions. All tankers
in the harbor require the services of a tug to move
into (and later out of) a berth. The tugs have speci ed priorities for beginning deberthing=berthing activities, based on their location (harbor, berths), the
availability of berths, and the queues of boats requiring berthing or deberthing. The area experiences
frequent storms. Tugs will not start new activities
when a storm is in progress but will always nish
an existing activity. If a tug is traveling from the
berths to the harbor without a tanker when a storm
begins, it will turn around and head for the berths.
The data collection unit we use is a single week. The
performance measure is the average time a ship waits
before beginning loading. As before, we investigate
the impact of the allocation ratio Ns =N , the total sample size N , and the driving frequency assignment on
the FDE factor screening ability.
3.1. Experimental design
Levels for the eleven factors we examine are given
in Table 2. We ran the experiments with three values
of N : N = 5000 (small), N = 10 000 (medium), and
N = 25 000 (large). Three sets of driving frequency
assignments (FA1 , FA2 , and FA3 ) are also used. Each
factor in FA1 is assigned a frequency using Jacobson
et al.’s algorithm [2] and the code provided in [11].
This frequency assignment results in unique identi cation frequencies for all main e ects, quadratic
e ects, and two-way interactions. The driving frequencies are rotated for FA2 and FA3 so that each
factor is oscillated at low, medium, and high frequencies during the course of our investigation. Our
window size for a particular signal (or noise) run is
set to min{2000; Ns − 100; (N − Ns ) − 100}. If the
Tukey window [1, p. 114] gives negative estimates
for noise runs, then we compute the absolute value of
negative signal-to-noise ratios. It took over 13 CPU
days on a Pentium 166 to conduct these 45 000 FDEs
– 1000 replications for each of the 45 combinations
of N , Ns =N , and frequency assignments. (Note that
all factor levels are dynamically oscillated within
each replication.) Files containing spectral di erences
and spectral ratios amounted to 165 megabytes of
summary data.
3.2. Baseline determination
In order to compare and validate FDE factor screening results, we should know the factors with greatest
impact on ship waiting time. Unlike the simple autoregressive system example of Section 3, we do not know
the true nature of the underlying response surface.
Therefore, we ran several full 311 (177 147 run) factorial experiments. Each of the 177 147 design points
corresponds to xing each of the eleven factors at
one of three levels. For quantitative factors, these are
the low and high levels given in Table 2, as well as
the center value 0.5(low level + high level). For the
ve qualitative factors (distribution types), the levels
correspond to using only distribution 1, only distribution 2, and an equally weighted mix of the distributions during the course of a simulation run.
We reran the factorial experiment several times, using run lengths ranging from N = 100 to N = 10 000.
(Shorter run lengths than those needed for frequency
domain experiments are possible because the factor
levels are xed during the course of each run: it is
the between-run performance di erence which indicates the presence of signi cant factor e ects.) Some
of the design points appear to be unstable (i.e., large
average queue lengths result). Nonetheless, regression
models involving eleven main e ects, 55 two-way interactions, and eleven quadratic e ects consistently
identify factor 3 (ship interarrival mean), factor 6
(number of berths), and factor 7 (loading time mean)
as the most important determinants of ship waiting
time, regardless of run length. (R2 ranges from 0.740
to 0.808 for models containing terms involving only
these factors, while R2 ≈ 0:813 for the full 77 term
models). We thus feel comfortable specifying that a
model with terms involving factors 3, 6 and 7 captures
the underlying ‘true’ relationship between the factors
and the waiting time.
3.3. FDE results
The results of the frequency domain experiments for
the African tanker example are summarized in Table 3.
After the spectral ratios and di erences are computed
87
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
Table 2
Factors and driving frequencies for the African tanker FDE experiments
ID
Factor name
1
2
3
4
5
6
7
8
9
Type
Storm time mean
Storm time distribution
Ship interarrival mean
Ship interarrival distribution
Number of tugs
Number of berths
Loading time meanb
Loading time distribution
Ship type distribution
Continuous
Qualitative
Continuous
Qualitative
Discrete
Discrete
Continuous
Qualitative
Qualitative
10
11
Calm time mean
Calm time distribution
Continuous
Qualitative
a Divide
entry by 508 to obtain f in cycles=observation.
is 18, 24, and 36 h, respectively, for tankers of type 1, 2, and 3.
b
Frequency assignmentsa
Factor levels
3–5 h
Uniform=exponential
8–14 h
Uniform=exponential
1–3
3–5
shiptype ± 8 h
Uniform=exponential
Multinomial with probability vector
{0:5; 0:25; 0:25} or {0:25; 0:25; 0:5}
24 –72 h
Uniform=exponential
FA1
FA2
FA3
1
4
10
17
29
52
67
89
132
89
132
164
205
1
4
10
17
29
17
29
52
67
89
132
164
205
1
164
205
52
67
4
10
shiptype
Table 3
African tanker results: identi cation of terms involving factors 3, 6 and 7
Ns =N
Terms
Percentage identi ed in top 12 termsa
Spectral ratios
N
Spectral di erences
N
5000
10 000
25 000
5000
10 000
25 000
0.1
Main e ects
Two-way interactions
Quadratic e ects
85.7
35.5
17.0
95.4
55.8
31.0
99.4
65.3
39.8
86.4
31.9
24.4
88.9
43.4
33.4
88.9
50.0
40.1
0.3
Main e ects
Two-way interactions
Quadratic e ects
93.5
50.7
26.9
98.4
62.9
38.5
99.9
66.5
41.4
88.9
47.1
36.2
87.7
50.5
41.1
92.1
53.0
43.8
0.5
Main e ects
Two-way interactions
Quadratic e ects
90.7
45.2
24.2
98.0
61.3
37.0
99.9
66.5
42.5
88.9
49.5
39.3
89.8
51.9
42.9
93.6
53.7
44.3
0.7
Main e ects
Two-way interactions
Quadratic e ects
88.1
41.7
22.6
94.2
51.9
28.5
99.7
65.7
41.8
87.3
50.6
40.7
91.0
52.5
43.5
94.3
54.1
44.4
0.9
Main e ects
Two-way interactions
Quadratic e ects
91.0
44.3
24.3
89.4
42.4
23.6
91.3
45.6
25.0
88.4
51.3
41.6
90.8
52.7
43.9
93.5
54.1
44.4
a Out
of 3000 experiments.
88
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
for 254 frequencies ({1=508; 2=508; : : : ; 254=508}) using the fast Fourier transformation code available in
[11], spikes corresponding to the 132 indicator frequencies (11 each for main and quadratic e ects, 110
for interaction terms) are ranked from highest to lowest magnitude. We de ne a term to be ‘identi ed’ for
a particular replication if it is among the top 12 spectral ratios (or di erences). The entries in Table 3 correspond to the percent of time any of the ‘correct’
terms (main, interaction, or quadratic e ects involving factors 3, 6 and 7) are among the top 12 indicator
frequency spikes in the experiment. Percentage values
which di er by more than 2.5 are statistically signi cant at level = 0:05.
Not surprisingly, in nearly all cases where Ns =N
is xed, increasing N increases the percent of terms
correctly identi ed. The exceptions are the allocation Ns =N = 0:9 for spectral ratios, and Ns =N = 0:3
for the main e ects using spectral di erences, where
there are no statistically signi cant increases or decreases. Thus, additional information generally improves the spectral identi cation. The spectral ratio
approach tends to be better at correctly identifying the
main e ects, while the di erence approach appears
better at identifying quadratic terms.
The results also indicate that the data allocation ratio a ects factor screening. Consider rst the spectral
ratio tests. When data are limited (N 610 000), the
best identi cation of main e ects, two-way interaction
e ects, or quadratic e ects occurs for an allocation
Ns =N = 0:3. However, the impact of Ns =N decreases
as N increases to 25 000, unlike our experience with
the autoregressive system in Section 2. It appears that
if the spectral ratio approach is used for evaluating
complex queueing systems, it is better to gain a good
estimate of the noise spectrum than to use an equal
allocation strategy. The results are not monotonic, so
there are limitations on how far this strategy can be
extended. Small run lengths for min{Ns ; N −Ns } when
N = 5000 and Ns =N 6= 0:5 indicate that the window sizes we use to estimate either the signal (or
noise) spectra are less than recommended [11], which
may contribute to the degradation in the procedure’s
performance.
For the spectral di erence approach, the best allocation strategy is not as consistent. For all three values of
N , the interaction and quadratic terms have the highest likelihood of appearing among the top 12 terms if
the allocation ratio is Ns =N = 0:9. For main e ects,
an allocation of 0.3 or 0.5 yields the largest observed
correct identi cation percentage when N = 5000, and
an allocation of 0.7 yields the largest results when
N = 10 000 or 25 000, although none of these results
are statistically di erent from the 0.9 allocation (at
level = 0:05). For xed total sample size N , the
best allocation for the ratio approach dominates that
of the best allocation for the di erence approach in
terms of the correct identi cation of the three main effects terms, while the di erence approach dominates
for identifying the quadratic e ects.
The African tanker experiments provide some interesting insight into the bene ts of long noise runs
as well, although further work is needed before generalizations can be made. Table 3 shows that if the
signal run length is xed, then increasing the length
of the noise run improves the screening ability of the
spectral ratio approach, though this cannot be said regarding the spectral di erence approach. For example,
compare the spectral ratio cells for {Ns =N = 0:1; N =
25 000} and {Ns =N =0:5; N =5000}. Both are based on
signal runs with 2500 observations, yet the detection
percentage is signi cantly higher (at = 0:05) in the
former case for main e ects, two-way interactions,
and quadratic e ects. While we do not have direct
comparability of signal run lengths in other situations,
there are a total of 14 combinations that can be made
with {Ns′ 6Ns ; N ′ ¿ N }. The 42 spectral ratio entries
in Table 3 corresponding to the main e ects, two-way
interactions and quadratic terms for these 14 combinations are always greater for {Ns′ ; N ′ } than for {Ns ; N }
and statistically signi cant (at = 0:05) in 88% of
the cases. In contrast, no clear pattern emerges for
the spectral di erence approach. It does appear that
if {Ns − Ns′ ¿1500}, then using {Ns ; N } instead of
{Ns′ ; N ′ } signi cantly increases the correct identi cation percentage for interaction and quadratic terms,
although not for main e ects.
4. Concluding remarks
Our results are not intended to be complete, but to
illustrate that an equal allocation of data between signal and noise runs probably will not result in the most
powerful FDE screening test. For simple simulation
systems with data constraints, the performance can
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
be improved by allocating the majority of the data to
signal runs (Ns =N ¿ 0:5). It may be that performance
improves further if spectral terms are pooled to form
the denominator used in the spectral ratio test, as suggested by Morrice and Schruben [5].
At times, the analyst might have baseline results
from a previous experiment or (if FDE is used on a
real system) historical operating data. If these data
are deemed adequate representations of system performance, most or all of the new e ort could be expended on the signal run. This might be particularly
bene cial for investigations of complex systems, such
as the African tanker problem examined in Section 3.
An interesting question is whether or not it would be
bene cial to use a single noise run for multiple signal runs if more than one signal run is to be made,
e ectively combining the results of three noise runs.
In the spectral ratio case, we observe that long noise
runs increase the ecacy of the screening procedure,
but it is not clear what would be lost by removing the
independence across frequency assignments.
Our results also indicate potential pitfalls. The investigation of autoregressive models in Section 2 indicates that when N is small, the spectral ratio method is
prone to large (spurious) spikes appearing near f= 0:5
or f = 0 cycles=observation. Although this behavior
is rapidly alleviated as the sample size increases, it
suggests that the analyst may wish to avoid frequencies close to either endpoint when N is limited. Since
Jacobson et al.’s algorithm [2] always speci es a frequency of the form 1=x for some integer divisor x,
the analyst might do better to prompt the procedure
for k + 1 driving frequencies (rather than k) and discard this lowest frequency, particularly if only a single run is to be made. In contrast, false positives for
the spectral di erence method appeared to occur over
wide bands of frequencies, indicating that this technique may be less suitable for screening purposes if
N is small. An issue that could be addressed in future
research is whether or not it is bene cial to use both
spectral ratios and spectral di erences within a single
FDE, since neither method dominates the other over
all driving frequencies.
The goal of this paper is to examine the e ects of
data allocation on the screening capabilities of frequency domain experiments, rather than to contrast
FDE results with those of conventional experimental
designs. Nonetheless, the African tanker results pro-
89
vide a vivid reminder of the need for this type of
screening procedure. It took just under 24 h of CPU
time on a Pentium 166 PC to generate the data for
the factorial experiment with N = 5000, with many
design points indicating an unstable system con guration. In contrast, a single replication of an FDE with
N = 25 000 required only 40 CPU seconds.
References
[1] C. Chat eld, The Analysis of Time Series, 4th Edition,
Chapman & Hall, New York, 1989.
[2] S.H. Jacobson, A.H. Buss, L.W. Schruben, Driving frequency
selection for frequency domain simulation experiments, Oper.
Res. 39 (6) (1991) 917–924.
[3] A.M. Law, W.D. Kelton, Simulation Modeling and Analysis,
2nd Edition, McGraw-Hill, New York, 1991.
[4] D.J. Morrice, S.H. Jacobson, Amplitude selection in
transient sensitivity analysis, in: C. Alexopoulos, K. Kang,
W.R. Lilegdon, D. Goldsman (Eds.), Proceedings of the 1995
Winter Simulation Conference, Institute of Electrical and
Electronic Engineers, Piscataway, NJ, 1995, pp. 330 –335.
[5] D.J. Morrice, L.W. Schruben, Simulation factor screening
using harmonic analysis, Management Sci. 39 (12) (1993)
1459–1476.
[6] A.A.B. Pritsker, Introduction to Simulation and SLAM II,
2nd Edition, Wiley, New York, 1989.
[7] J.K. Robinson, L.W. Schruben, J.W. Fowler, Experimentation
with large-scale semiconductor simulations: a frequency
domain approach to factor screening, in: D. Mitta, L. Burke,
G. Tonkay, J. English, J. Galliamore, G. Klutke (Eds.),
Proceedings of the 1993 IIE Research Conference, Institute
of Industrial Engineers, Los Angeles, CA, 1993.
[8] P.J. Sanchez, Design and analysis of frequency domain
experiments, Ph.D. Dissertation, School of Operations
Research and Industrial Engineering, Cornell University,
Ithaca, NY, 1987.
[9] P.J. Sanchez, A.H. Buss, A model for frequency domain
experiments, in: A. Thesen, H. Grant, W.D. Kelton (Eds.),
Proceedings of the 1987 Winter Simulation Conference,
Institute of Electrical and Electronic Engineers, Piscataway,
NJ, pp. 424 – 427.
[10] P.J. Sanchez, S.M. Sanchez, Design of frequency-domain
experiments for discrete-valued factors, Appl. Math. Comput.
42 (1) (1991) 1–21.
[11] S.M. Sanchez, F. Moeeni, P.J. Sanchez, So many factors, so
little time: a frequency domain approach for factor screening,
Technol. Oper. Rev. (2000), forthcoming.
[12] R.G. Sargent, T.K. Som, Current issues in frequency domain
experimentation, Management Sci. 38 (1992) 667–687.
[13] T.J. Schriber, Simulation Using GPSS, Wiley, New York,
1974.
[14] L.W. Schruben, V.J. Cogliano, An experimental procedure for
simulation response surface identi cation, Commun. ACM
30 (8) (1987) 716–730.
www.elsevier.com/locate/orms
Ecient data allocation for frequency domain experiments (
Susan M. Sanchez a; ∗ , Prabhudev Konanab
a Management
Science and Information Systems, University of Missouri-St. Louis, School of Business Administration,
St. Louis, MO 63121-4499, USA
b Department of Management Science and Information Systems, The University of Texas at Austin, USA
Received 1 July 1995; received in revised form 1 September 1999
Abstract
Frequency domain experiments are ecient screening procedures for identifying important factors in simulation models.
An empirical investigation involving a simple autoregressive system and a complex queueing system shows that allocating
data unequally among the signal and noise runs may be more e ective when the total data collection e ort is limited.
c 2000 Elsevier Science B.V. All rights reserved.
Keywords: Simulation; Design of experiments; Factor screening
1. Introduction
One of the goals of many simulation studies is to understand the relationships among various input factors
and how their levels a ect the system response. The
simplicity of factorial designs makes them attractive
to analysts interested in examining systems characterized by a moderately small number of factors or simple (linear) response relationships. However, despite
the recent improvements in computing power and
cost, the time required to conduct a thorough factorial
analysis of a simulated system may be prohibitive if
the number of input factors is large. If output data are
highly autocorrelated, then large portions may need to
( Supported in part by the University of Missouri-St. Louis
Oce of Research, and Grant DDM-9396135 from the National
Science Foundation.
∗ Corresponding author.
be truncated from each run in order to remove initialization bias e ects. Further, non-linear model structures with non-constant variance may also be needed
to adequately model a complex system’s behavior
and arrive at statistically valid assessments of the
system.
Schruben and Cogliano [14] proposed frequency
domain experimentation (FDE) as an alternative to traditional factor screening techniques in computer simulation (see also the work of Sanchez and Buss [9]).
FDE provides a powerful and convenient method for
simulation factor screening with just two runs: a noise
run and a signal run. The noise run acts as a control for any natural cyclic behavior of the system and
is generated by holding all factors at nominal levels.
The signal run is generated by sinusoidally oscillating
the factor levels, each at a di erent driving frequency,
f. These driving frequencies are chosen such that
the output terms of polynomial order correspond to
c 2000 Elsevier Science B.V. All rights reserved.
0167-6377/00/$ - see front matter
PII: S 0 1 6 7 - 6 3 7 7 ( 9 9 ) 0 0 0 7 0 - X
82
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
identi able indicator frequencies on the spectrum. For
binary factors, the probability of achieving a speci ed
level is oscillated sinusoidally. This approach extends
to qualitative factors with three or more levels via a
binary tree structure; levels for quantitative discrete
factors can also be approximated by continuous sine
functions [10].
Once the signal and noise runs have been generated,
signal and noise spectra can be computed using (fast)
Fourier transformations. The existence of important
factor e ects can then be assessed by examining the
spikes in the signal distribution at the corresponding
indicator frequencies. There are two approaches to
analyzing these spikes: the spectral ratio approach
[14] and the spectral di erence approach. In the
spectral ratio approach a factor is said to a ect the response if the ratio of the signal spectrum to the noise
spectrum at its identifying frequency is substantially
greater than one. The spectral ratio asymptotically
follows an F distribution. In the spectral di erence
approach [7,12], the noise spectral terms are subtracted from the signal spectral terms (by frequency)
and the resulting spectral di erences are asymptotically distributed 2 . In both situations, a qualitative
rather than quantitative approach is recommended
since the degrees of freedom associated with the F
or 2 distributions must be approximated when the
noise terms are correlated. (For a detailed explanation of FDE, along with code necessary to select the
driving frequencies and perform the Fourier transformations, see the on-line or o -line version of Sanchez
et al. [11].)
In past studies, data were allocated equally between
the noise and signal runs, more for convenience than
for any scienti c reason. How do allocation strategies
a ect the spectral tests? Will increasing the proportion
of data to the signal (or noise) run result in a greater
screening ability? Is there a desirable (optimal) absolute data amount for signal and noise runs? These
questions are of interest when experiments must be
conducted under xed budget or time constraints. In
this paper, we analyze the power of spectral ratio and
di erence tests under various data allocation strategies. Sections 2 and 3 contain empirical results from
a simple autoregressive system and 11-factor queueing system, respectively. We identify pitfalls, propose
preliminary guidelines for data allocation, and discuss
future research directions in Section 4.
2. Autoregressive system example
Consider the following single-factor linear model:
p
Yt = Xt + et ; et = 1 − 2 Zt + et−1
where Yt ; Xt and et are the response from the simulation model, the input factor level, and the noise term,
respectively, at time t. The Zt ’s are i.i.d. standard normal random variates. Xt is held at a nominal level of
zero during the noise run. During the signal run, Xt is
oscillated at Xt = m cos(2ft), where f is the driving
frequency in cycles per observation, and m is the amplitude of the oscillation. The et ’s are independent of
the input factor Xt , but are positively (or negatively)
autocorrelated depending on the value of the correlation coecient . To facilitate comparisons, the scaling factor is included so that the error variance is equal
to one for all ; thus m is the ratio of the factor e ect
to the error standard deviation.
2.1. Experimental design
For this simple linear system, there are ve factors
that might a ect the ability of the FDE to identify the
presence of the Xt term in the model: (1) the amplitude
of the input signal oscillation, (2) the autocorrelation
of the error distribution, (3) the total sample size, (4)
the driving frequency for the input signal, and (5) the
proportion of the total sample allocated to the signal
run. Of these, the rst two factors are characteristics
of the system itself. The oscillation amplitude plays
an important role in the factor screening process [4],
while the error correlation structure impacts the shape
of the noise spectrum. For the simple linear system we
study, the noise spectrum is
at if = 0, has power
concentrated at low frequencies (skewed right) for
¿ 0 and has power concentrated at high frequencies
(skewed left) for ¡ 0.
The nal three factors are speci ed by the experimenter. Let N denote the total sample size and
Ns denote the sample size allocated to the signal
run. The driving frequency for our single input factor can be speci ed at any value between 0 and
0.5 cycles=observation, although for more complex
modeling situations involving many input factors the
driving frequencies cannot be chosen with complete
freedom ([2], see [11] for code). Spectral ratio tests
will be more powerful if the driving frequencies are
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
those for which the noise spectral values (i.e., system gain) are low [14], but guidelines for allocation
should ideally be robust to the speci cation of driving
frequency values. (Often the analyst may not know
the system gain behavior a priori, and it may not be
possible to oscillate all factors at frequencies that
correspond to low noise spectral ratios during investigations of complex systems involving many factors.)
Our primary questions of interest are the e ect of
Ns =N on factor screening capability, and whether or
not this varies with N . Our experimental settings for
the ve factors are:
• Signal amplitude (m): 0:1; 0:5
• Correlation coecient (): − 0:9 to +0:9 (by increment of 0.2)
• Proportion of data allocated to signal (Ns =N ): 0:1
to 0.9 (by increment of 0.2)
• Total sample size (N ): 1000; 5000 and 10 000
• Driving frequency (f, in cycles=observation):
0:1; 0:4
Each combination of the factor levels (600 in all)
is a design point. 1000 independently seeded replications of the FDE are generated for each design point,
with signal and noise runs independently seeded. This
same set of random number seeds is common to all
design points for increased eciency in comparisons.
Fourier transforms are evaluated at the frequencies
f = 0(0:01)0:5 using a Tukey window [1, p. 114].
2.2. Results
We rst illustrate the performance of the method in
terms of overall identi cation ability. Let Imax denote
the number of times (out of the 1000 replications)
that the driving frequency yielded the largest spectral
ratio (or di erence) among the 51 frequencies evaluated. Fig. 1 graphs the results of our experiment
when the driving frequency f = 0:1 and amplitude
m = 0:1; 0:5 (the shaded surface corresponds to 0.1
and the unshaded corresponds to 0.5) for both spectral
ratios and spectral di erences. Each subgraph shows
Imax as a function of the allocation proportion Ns =N
and the error correlation . From these graphs, we see
that spectral ratios have the greatest power (i.e., the
highest probability of correctly detecting the model
term) for far from zero, and spectral di erences have
83
the greatest power for near zero. As expected, the
power increases as the sample size N increases or as
the amplitude of the factor level increases. Graphs for
f = 0:4 (not shown) essentially re
ect the curves of
Fig. 1 around = 0.
The allocation proportion clearly a ects the power
of both spectral tests, although the results are most
readily apparent for the case m = 0:1. For the most
part, the screening power is the lowest at Ns =N = 0:1;
the power increases as Ns =N increases to 0.7. Any
further increase in Ns =N results in lower or equal
spectral power for the spectral ratio approach, while
the power increases slightly for some of the spectral
di erence experiments. Similar gures result when
the driving frequency is changed. Thus, it appears
that identi cation can be improved for small-sample
FDEs by allocating more than half of the data to the
signal run.
We also consider the performance on an individual
experiment basis by examining the number of identi cations as functions of the data allocation and spectral frequency for a subgroup of the design points:
N = 1000 or 5000; = 0:7 and f = 0:1. We use this
subgroup for illustration because, as Fig. 1 indicates,
neither spectral approach does a good job of correctly
identifying the indicator frequency across all allocation proportions and both values of m. When spectral
ratios are used, then for low N and low m the frequencies most likely to be associated with the highest
spectral ratio are 0.0 and 0.5: the indicator frequency
is not selected particularly more than any of the other
frequencies, although the corresponding Imax is lowest
for Ns =N = 0:1. As the sample size increases, the indicator frequency is much more likely to yield the largest
spike: only the frequency of 0.5 also stands out against
the background noise. For the larger value of m, the
indicator frequency is chosen most often. Neighboring frequencies occasionally yield larger spectral ratios for small Ns =N , due to a phenomenon known as
the smearing of the spectrum. A pattern of screening power that increases sharply and then decreases
slightly with Ns =N occurs for N = 1000; the procedure
correctly identi es the indicator frequency in all cases
for Ns =N ¿0:3 when N = 5000.
The individual performance curves for spectral
di erences di er markedly from those for spectral
ratios. When m is small, false positives over a band
of low frequencies dominate the indicator frequency
84
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
Fig. 1. Correct identi cations for autoregressive system, amplitude m = 0:1 (shaded) and m = 0:5 (unshaded).
spike. This occurs, regardless of the driving frequency, because the noise spectrum is skewed right
( ¿ 0). If ¡ 0, then the false positives are clustered around frequencies near 0.5 rather than those
near zero. However, consistent with Fig. 1, the identi cation power increases and then stabilizes as Ns =N
increases.
A more detailed look at the performance is provided
in Table 1, in which terms have been averaged over
runs with non-identical factors in order to reduce the
impact of inherent system characteristics. To construct
this table we begin, for every one of the 600 design
points, by computing the mean and standard deviation of the spectral ratios (or di erences) for each of
51 frequencies (0.0 to 0.5 by 0.01) over 1000 replications. The results for the indicator frequencies are
further averaged over and m to create entries of Table 1 for the spike means and standard deviations. All
spectral ratios (or di erences) for the 50 non-indicator
frequencies have been averaged over and m, and
reported as the background means and standard deviations in this table. The results provide a robust
view of the e ects of data allocation on the screening
capability.
85
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
Table 1
Average magnitudes of spectral terms for autoregressive system
Ns =N
Frequency
Indicator
0.1
Background
Indicator
0.3
Background
Indicator
0.5
Background
Indicator
0.7
Background
Indicator
0.9
Background
Statistic
Spectral ratios
N
Spectral di erences
N
1000
5000
10 000
1000
Mean
Std. dev.
Mean
Std. dev.
4.54
1.92
1.23
0.72
Mean
Std. dev.
Mean
Std. dev.
8.43
2.02
1.10
0.41
11.29
2.15
1.04
0.33
0.31
0.24
−0:00
0.20
0.69
0.21
0.00
0.13
0.97
0.20
−0:00
0.11
7.06
2.64
1.21
0.60
13.91
3.08
1.03
0.34
19.01
3.43
1.03
0.28
0.53
0.23
0.01
0.16
1.20
0.20
−0:00
0.11
1.70
0.19
−0:00
0.09
Mean
Std. dev.
Mean
Std. dev.
9.12
3.51
1.20
0.60
17.84
4.13
1.05
0.33
24.40
4.66
1.03
0.27
0.69
0.22
0.01
0.16
1.55
0.20
0.00
0.11
2.19
0.19
−0:00
0.09
Mean
Std. dev.
Mean
Std. dev.
11.00
5.05
1.22
0.66
21.16
5.55
1.07
0.36
28.88
6.17
1.05
0.29
0.82
0.23
0.01
0.16
1.83
0.20
0.00
0.11
2.59
0.19
0.00
0.09
Mean
Std. dev.
Mean
Std. dev.
14.67
18.33
1.42
2.90
25.17
8.99
1.12
0.47
33.77
9.68
1.09
0.37
0.93
0.25
0.02
0.20
2.08
0.21
0.00
0.13
2.94
0.20
0.00
0.11
Because the term associated with the indicator
frequency may be selected as important even if it
does not yield the largest spectral ratio (or di erence), Table 1 provides some information about how
powerful the screening procedure might be for an
unknown (simple) system. For example, the mean
spectral ratio associated with the indicator frequency
is 3.5 times as large as the mean background ratio
when N = 1000 and Ns =N = 0:1. When Ns =N increases to 0.9 for this value of N , the mean spectral
ratio for the indicator frequency is 10 times as large
as the mean background ratio. This improvement
(roughly tripling the average spike for the indicator
frequency) also holds as Ns =N increases from 0.1 to
0.9 for the other two values of N , even though the
tests with larger N are initially more powerful. For
N = 10 000, the mean spectral ratio for the indicator
is about 11 times as large as the mean background
ratio when Ns =N = 0:1, and about 33 times as large
5000
10 000
as the mean background ratio when Ns =N = 0:9. The
spectral di erences show similar behavior, although
the scale is di erent. The average spectral di erences for the non-indicator frequencies conform with
theory, and are not statistically distinguishable from
zero. Increasing the allocation from Ns =N = 0:1 to
Ns =N = 0:9 roughly triples the magnitude of the spike
associated with the indicator frequency. This shows
that the power of both the spectral ratio and di erence
approaches improves as the allocation ratio increases,
at least over the range of allocations and sample sizes
examined.
3. African tanker example
Our second example is the African tanker problem
[3]. This problem has also been studied by Pritsker [6],
Schriber [13], and Sanchez [8]. A brief description of
the system follows.
86
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
A port in Africa is used to load tankers with crude
oil for overwater shipment, and the port has facilities for loading more than one tanker simultaneously. The tankers are of three di erent types and
have di erent service time distributions. All tankers
in the harbor require the services of a tug to move
into (and later out of) a berth. The tugs have speci ed priorities for beginning deberthing=berthing activities, based on their location (harbor, berths), the
availability of berths, and the queues of boats requiring berthing or deberthing. The area experiences
frequent storms. Tugs will not start new activities
when a storm is in progress but will always nish
an existing activity. If a tug is traveling from the
berths to the harbor without a tanker when a storm
begins, it will turn around and head for the berths.
The data collection unit we use is a single week. The
performance measure is the average time a ship waits
before beginning loading. As before, we investigate
the impact of the allocation ratio Ns =N , the total sample size N , and the driving frequency assignment on
the FDE factor screening ability.
3.1. Experimental design
Levels for the eleven factors we examine are given
in Table 2. We ran the experiments with three values
of N : N = 5000 (small), N = 10 000 (medium), and
N = 25 000 (large). Three sets of driving frequency
assignments (FA1 , FA2 , and FA3 ) are also used. Each
factor in FA1 is assigned a frequency using Jacobson
et al.’s algorithm [2] and the code provided in [11].
This frequency assignment results in unique identi cation frequencies for all main e ects, quadratic
e ects, and two-way interactions. The driving frequencies are rotated for FA2 and FA3 so that each
factor is oscillated at low, medium, and high frequencies during the course of our investigation. Our
window size for a particular signal (or noise) run is
set to min{2000; Ns − 100; (N − Ns ) − 100}. If the
Tukey window [1, p. 114] gives negative estimates
for noise runs, then we compute the absolute value of
negative signal-to-noise ratios. It took over 13 CPU
days on a Pentium 166 to conduct these 45 000 FDEs
– 1000 replications for each of the 45 combinations
of N , Ns =N , and frequency assignments. (Note that
all factor levels are dynamically oscillated within
each replication.) Files containing spectral di erences
and spectral ratios amounted to 165 megabytes of
summary data.
3.2. Baseline determination
In order to compare and validate FDE factor screening results, we should know the factors with greatest
impact on ship waiting time. Unlike the simple autoregressive system example of Section 3, we do not know
the true nature of the underlying response surface.
Therefore, we ran several full 311 (177 147 run) factorial experiments. Each of the 177 147 design points
corresponds to xing each of the eleven factors at
one of three levels. For quantitative factors, these are
the low and high levels given in Table 2, as well as
the center value 0.5(low level + high level). For the
ve qualitative factors (distribution types), the levels
correspond to using only distribution 1, only distribution 2, and an equally weighted mix of the distributions during the course of a simulation run.
We reran the factorial experiment several times, using run lengths ranging from N = 100 to N = 10 000.
(Shorter run lengths than those needed for frequency
domain experiments are possible because the factor
levels are xed during the course of each run: it is
the between-run performance di erence which indicates the presence of signi cant factor e ects.) Some
of the design points appear to be unstable (i.e., large
average queue lengths result). Nonetheless, regression
models involving eleven main e ects, 55 two-way interactions, and eleven quadratic e ects consistently
identify factor 3 (ship interarrival mean), factor 6
(number of berths), and factor 7 (loading time mean)
as the most important determinants of ship waiting
time, regardless of run length. (R2 ranges from 0.740
to 0.808 for models containing terms involving only
these factors, while R2 ≈ 0:813 for the full 77 term
models). We thus feel comfortable specifying that a
model with terms involving factors 3, 6 and 7 captures
the underlying ‘true’ relationship between the factors
and the waiting time.
3.3. FDE results
The results of the frequency domain experiments for
the African tanker example are summarized in Table 3.
After the spectral ratios and di erences are computed
87
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
Table 2
Factors and driving frequencies for the African tanker FDE experiments
ID
Factor name
1
2
3
4
5
6
7
8
9
Type
Storm time mean
Storm time distribution
Ship interarrival mean
Ship interarrival distribution
Number of tugs
Number of berths
Loading time meanb
Loading time distribution
Ship type distribution
Continuous
Qualitative
Continuous
Qualitative
Discrete
Discrete
Continuous
Qualitative
Qualitative
10
11
Calm time mean
Calm time distribution
Continuous
Qualitative
a Divide
entry by 508 to obtain f in cycles=observation.
is 18, 24, and 36 h, respectively, for tankers of type 1, 2, and 3.
b
Frequency assignmentsa
Factor levels
3–5 h
Uniform=exponential
8–14 h
Uniform=exponential
1–3
3–5
shiptype ± 8 h
Uniform=exponential
Multinomial with probability vector
{0:5; 0:25; 0:25} or {0:25; 0:25; 0:5}
24 –72 h
Uniform=exponential
FA1
FA2
FA3
1
4
10
17
29
52
67
89
132
89
132
164
205
1
4
10
17
29
17
29
52
67
89
132
164
205
1
164
205
52
67
4
10
shiptype
Table 3
African tanker results: identi cation of terms involving factors 3, 6 and 7
Ns =N
Terms
Percentage identi ed in top 12 termsa
Spectral ratios
N
Spectral di erences
N
5000
10 000
25 000
5000
10 000
25 000
0.1
Main e ects
Two-way interactions
Quadratic e ects
85.7
35.5
17.0
95.4
55.8
31.0
99.4
65.3
39.8
86.4
31.9
24.4
88.9
43.4
33.4
88.9
50.0
40.1
0.3
Main e ects
Two-way interactions
Quadratic e ects
93.5
50.7
26.9
98.4
62.9
38.5
99.9
66.5
41.4
88.9
47.1
36.2
87.7
50.5
41.1
92.1
53.0
43.8
0.5
Main e ects
Two-way interactions
Quadratic e ects
90.7
45.2
24.2
98.0
61.3
37.0
99.9
66.5
42.5
88.9
49.5
39.3
89.8
51.9
42.9
93.6
53.7
44.3
0.7
Main e ects
Two-way interactions
Quadratic e ects
88.1
41.7
22.6
94.2
51.9
28.5
99.7
65.7
41.8
87.3
50.6
40.7
91.0
52.5
43.5
94.3
54.1
44.4
0.9
Main e ects
Two-way interactions
Quadratic e ects
91.0
44.3
24.3
89.4
42.4
23.6
91.3
45.6
25.0
88.4
51.3
41.6
90.8
52.7
43.9
93.5
54.1
44.4
a Out
of 3000 experiments.
88
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
for 254 frequencies ({1=508; 2=508; : : : ; 254=508}) using the fast Fourier transformation code available in
[11], spikes corresponding to the 132 indicator frequencies (11 each for main and quadratic e ects, 110
for interaction terms) are ranked from highest to lowest magnitude. We de ne a term to be ‘identi ed’ for
a particular replication if it is among the top 12 spectral ratios (or di erences). The entries in Table 3 correspond to the percent of time any of the ‘correct’
terms (main, interaction, or quadratic e ects involving factors 3, 6 and 7) are among the top 12 indicator
frequency spikes in the experiment. Percentage values
which di er by more than 2.5 are statistically signi cant at level = 0:05.
Not surprisingly, in nearly all cases where Ns =N
is xed, increasing N increases the percent of terms
correctly identi ed. The exceptions are the allocation Ns =N = 0:9 for spectral ratios, and Ns =N = 0:3
for the main e ects using spectral di erences, where
there are no statistically signi cant increases or decreases. Thus, additional information generally improves the spectral identi cation. The spectral ratio
approach tends to be better at correctly identifying the
main e ects, while the di erence approach appears
better at identifying quadratic terms.
The results also indicate that the data allocation ratio a ects factor screening. Consider rst the spectral
ratio tests. When data are limited (N 610 000), the
best identi cation of main e ects, two-way interaction
e ects, or quadratic e ects occurs for an allocation
Ns =N = 0:3. However, the impact of Ns =N decreases
as N increases to 25 000, unlike our experience with
the autoregressive system in Section 2. It appears that
if the spectral ratio approach is used for evaluating
complex queueing systems, it is better to gain a good
estimate of the noise spectrum than to use an equal
allocation strategy. The results are not monotonic, so
there are limitations on how far this strategy can be
extended. Small run lengths for min{Ns ; N −Ns } when
N = 5000 and Ns =N 6= 0:5 indicate that the window sizes we use to estimate either the signal (or
noise) spectra are less than recommended [11], which
may contribute to the degradation in the procedure’s
performance.
For the spectral di erence approach, the best allocation strategy is not as consistent. For all three values of
N , the interaction and quadratic terms have the highest likelihood of appearing among the top 12 terms if
the allocation ratio is Ns =N = 0:9. For main e ects,
an allocation of 0.3 or 0.5 yields the largest observed
correct identi cation percentage when N = 5000, and
an allocation of 0.7 yields the largest results when
N = 10 000 or 25 000, although none of these results
are statistically di erent from the 0.9 allocation (at
level = 0:05). For xed total sample size N , the
best allocation for the ratio approach dominates that
of the best allocation for the di erence approach in
terms of the correct identi cation of the three main effects terms, while the di erence approach dominates
for identifying the quadratic e ects.
The African tanker experiments provide some interesting insight into the bene ts of long noise runs
as well, although further work is needed before generalizations can be made. Table 3 shows that if the
signal run length is xed, then increasing the length
of the noise run improves the screening ability of the
spectral ratio approach, though this cannot be said regarding the spectral di erence approach. For example,
compare the spectral ratio cells for {Ns =N = 0:1; N =
25 000} and {Ns =N =0:5; N =5000}. Both are based on
signal runs with 2500 observations, yet the detection
percentage is signi cantly higher (at = 0:05) in the
former case for main e ects, two-way interactions,
and quadratic e ects. While we do not have direct
comparability of signal run lengths in other situations,
there are a total of 14 combinations that can be made
with {Ns′ 6Ns ; N ′ ¿ N }. The 42 spectral ratio entries
in Table 3 corresponding to the main e ects, two-way
interactions and quadratic terms for these 14 combinations are always greater for {Ns′ ; N ′ } than for {Ns ; N }
and statistically signi cant (at = 0:05) in 88% of
the cases. In contrast, no clear pattern emerges for
the spectral di erence approach. It does appear that
if {Ns − Ns′ ¿1500}, then using {Ns ; N } instead of
{Ns′ ; N ′ } signi cantly increases the correct identi cation percentage for interaction and quadratic terms,
although not for main e ects.
4. Concluding remarks
Our results are not intended to be complete, but to
illustrate that an equal allocation of data between signal and noise runs probably will not result in the most
powerful FDE screening test. For simple simulation
systems with data constraints, the performance can
S.M. Sanchez, P. Konana / Operations Research Letters 26 (2000) 81–89
be improved by allocating the majority of the data to
signal runs (Ns =N ¿ 0:5). It may be that performance
improves further if spectral terms are pooled to form
the denominator used in the spectral ratio test, as suggested by Morrice and Schruben [5].
At times, the analyst might have baseline results
from a previous experiment or (if FDE is used on a
real system) historical operating data. If these data
are deemed adequate representations of system performance, most or all of the new e ort could be expended on the signal run. This might be particularly
bene cial for investigations of complex systems, such
as the African tanker problem examined in Section 3.
An interesting question is whether or not it would be
bene cial to use a single noise run for multiple signal runs if more than one signal run is to be made,
e ectively combining the results of three noise runs.
In the spectral ratio case, we observe that long noise
runs increase the ecacy of the screening procedure,
but it is not clear what would be lost by removing the
independence across frequency assignments.
Our results also indicate potential pitfalls. The investigation of autoregressive models in Section 2 indicates that when N is small, the spectral ratio method is
prone to large (spurious) spikes appearing near f= 0:5
or f = 0 cycles=observation. Although this behavior
is rapidly alleviated as the sample size increases, it
suggests that the analyst may wish to avoid frequencies close to either endpoint when N is limited. Since
Jacobson et al.’s algorithm [2] always speci es a frequency of the form 1=x for some integer divisor x,
the analyst might do better to prompt the procedure
for k + 1 driving frequencies (rather than k) and discard this lowest frequency, particularly if only a single run is to be made. In contrast, false positives for
the spectral di erence method appeared to occur over
wide bands of frequencies, indicating that this technique may be less suitable for screening purposes if
N is small. An issue that could be addressed in future
research is whether or not it is bene cial to use both
spectral ratios and spectral di erences within a single
FDE, since neither method dominates the other over
all driving frequencies.
The goal of this paper is to examine the e ects of
data allocation on the screening capabilities of frequency domain experiments, rather than to contrast
FDE results with those of conventional experimental
designs. Nonetheless, the African tanker results pro-
89
vide a vivid reminder of the need for this type of
screening procedure. It took just under 24 h of CPU
time on a Pentium 166 PC to generate the data for
the factorial experiment with N = 5000, with many
design points indicating an unstable system con guration. In contrast, a single replication of an FDE with
N = 25 000 required only 40 CPU seconds.
References
[1] C. Chat eld, The Analysis of Time Series, 4th Edition,
Chapman & Hall, New York, 1989.
[2] S.H. Jacobson, A.H. Buss, L.W. Schruben, Driving frequency
selection for frequency domain simulation experiments, Oper.
Res. 39 (6) (1991) 917–924.
[3] A.M. Law, W.D. Kelton, Simulation Modeling and Analysis,
2nd Edition, McGraw-Hill, New York, 1991.
[4] D.J. Morrice, S.H. Jacobson, Amplitude selection in
transient sensitivity analysis, in: C. Alexopoulos, K. Kang,
W.R. Lilegdon, D. Goldsman (Eds.), Proceedings of the 1995
Winter Simulation Conference, Institute of Electrical and
Electronic Engineers, Piscataway, NJ, 1995, pp. 330 –335.
[5] D.J. Morrice, L.W. Schruben, Simulation factor screening
using harmonic analysis, Management Sci. 39 (12) (1993)
1459–1476.
[6] A.A.B. Pritsker, Introduction to Simulation and SLAM II,
2nd Edition, Wiley, New York, 1989.
[7] J.K. Robinson, L.W. Schruben, J.W. Fowler, Experimentation
with large-scale semiconductor simulations: a frequency
domain approach to factor screening, in: D. Mitta, L. Burke,
G. Tonkay, J. English, J. Galliamore, G. Klutke (Eds.),
Proceedings of the 1993 IIE Research Conference, Institute
of Industrial Engineers, Los Angeles, CA, 1993.
[8] P.J. Sanchez, Design and analysis of frequency domain
experiments, Ph.D. Dissertation, School of Operations
Research and Industrial Engineering, Cornell University,
Ithaca, NY, 1987.
[9] P.J. Sanchez, A.H. Buss, A model for frequency domain
experiments, in: A. Thesen, H. Grant, W.D. Kelton (Eds.),
Proceedings of the 1987 Winter Simulation Conference,
Institute of Electrical and Electronic Engineers, Piscataway,
NJ, pp. 424 – 427.
[10] P.J. Sanchez, S.M. Sanchez, Design of frequency-domain
experiments for discrete-valued factors, Appl. Math. Comput.
42 (1) (1991) 1–21.
[11] S.M. Sanchez, F. Moeeni, P.J. Sanchez, So many factors, so
little time: a frequency domain approach for factor screening,
Technol. Oper. Rev. (2000), forthcoming.
[12] R.G. Sargent, T.K. Som, Current issues in frequency domain
experimentation, Management Sci. 38 (1992) 667–687.
[13] T.J. Schriber, Simulation Using GPSS, Wiley, New York,
1974.
[14] L.W. Schruben, V.J. Cogliano, An experimental procedure for
simulation response surface identi cation, Commun. ACM
30 (8) (1987) 716–730.