07350015%2E2014%2E925807
Journal of Business & Economic Statistics
ISSN: 0735-0015 (Print) 1537-2707 (Online) Journal homepage: http://www.tandfonline.com/loi/ubes20
Numerically Accelerated Importance Sampling for
Nonlinear Non-Gaussian State-Space Models
Siem Jan Koopman, André Lucas & Marcel Scharth
To cite this article: Siem Jan Koopman, André Lucas & Marcel Scharth (2015) Numerically
Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models, Journal of
Business & Economic Statistics, 33:1, 114-127, DOI: 10.1080/07350015.2014.925807
To link to this article: http://dx.doi.org/10.1080/07350015.2014.925807
View supplementary material
Accepted author version posted online: 12
Jun 2014.
Submit your article to this journal
Article views: 242
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=ubes20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 19:19
Supplementary materials for this article are available online. Please go to http://tandfonline.com/r/JBES
Numerically Accelerated Importance Sampling
for Nonlinear Non-Gaussian State-Space
Models
Siem Jan KOOPMAN
Department of Econometrics, VU University Amsterdam, 1081 HV Amsterdam, The Netherlands;
Tinbergen Institute Amsterdam, The Netherlands; CREATES, Aarhus University, DK-8210 Aarhus, Denmark
(s.j.koopman@vu.nl)
Andre´ LUCAS
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
Department of Finance, VU University Amsterdam, 1081 HV Amsterdam, The Netherlands; Tinbergen Institute
and Duisenberg School of Finance, 1082 MS Amsterdam, The Netherlands (a.lucas@vu.nl)
Marcel SCHARTH
Australian School of Business, University of New South Wales, Kensington NSW 2052, Australia
(m.scharth@unsw.edu.au)
We propose a general likelihood evaluation method for nonlinear non-Gaussian state-space models using
the simulation-based method of efficient importance sampling. We minimize the simulation effort by
replacing some key steps of the likelihood estimation procedure by numerical integration. We refer to this
method as numerically accelerated importance sampling. We show that the likelihood function for models
with a high-dimensional state vector and a low-dimensional signal can be evaluated more efficiently using
the new method. We report many efficiency gains in an extensive Monte Carlo study as well as in an
empirical application using a stochastic volatility model for U.S. stock returns with multiple volatility
factors. Supplementary materials for this article are available online.
KEY WORDS: Control variables; Efficient importance sampling; Kalman filter; Numerical integration;
Simulated maximum likelihood; Simulation smoothing; Stochastic volatility model.
1.
INTRODUCTION
The evaluation of an analytically intractable likelihood function is a challenging task for many statistical and econometric
time series models. The key challenge is the computation of
a high-dimensional integral which is typically carried out by
importance sampling methods. Advances in importance sampling over the past three decades have contributed to the interest in nonlinear non-Gaussian state-space models that in
most cases lack a tractable likelihood expression. Examples
include stochastic volatility models as in Ghysels, Harvey, and
Renault (1996), stochastic conditional intensity models as in
Bauwens and Hautsch (2006), non-Gaussian unobserved components time series models as in Durbin and Koopman (2000),
and flexible nonlinear panel data models with unobserved heterogeneity as in Heiss (2008).
We propose a new numerically and computationally efficient
importance sampling method for nonlinear non-Gaussian statespace models. We show that a major part of the likelihood
evaluation procedure can be done by fast numerical integration rather than Monte Carlo integration only. Our main contribution consists of two parts. First, a numerical integration
scheme is developed to construct an efficient importance density that minimizes the log-variance of the simulation error. Our
approach is based on the efficient importance sampling (EIS)
method of Richard and Zhang (2007), which relies on Monte
Carlo simulations to obtain the importance density. Second, we
propose new control variables that eliminate the first order simulation error in evaluating the likelihood function via importance
sampling.
Numerical integration is generally highly accurate but its
feasibility is limited to low-dimensional problems. Although
the Monte Carlo integration method is applicable to high dimensional problems, it is subject to simulation error. These
properties are typical to applications in time series modeling.
Here we adopt both methods and we show how to carry the
virtues of numerical integration over to high dimensional statespace models. We depart from the numerical approaches of
Kitagawa (1987) and Fridman and Harris (1998) as well as from
the simulation based methods of Danielsson and Richard (1993)
and Durbin and Koopman (1997). We refer to our new method
as numerically accelerated importance sampling (NAIS).
Following Shephard and Pitt (1997) and Durbin and Koopman
(1997) (referred to as SPDK), we base our importance sampling
method on an approximating linear Gaussian state-space model
and use computationally efficient methods for this class of models. We explore two different, but numerically equivalent, sets
of algorithms. The first approach follows SPDK and considers
114
© 2015 American Statistical Association
Journal of Business & Economic Statistics
January 2015, Vol. 33, No. 1
DOI: 10.1080/07350015.2014.925807
Color versions of one or more of the figures in the article can be
found online at www.tandfonline.com/r/jbes.
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
Koopman, Lucas, and Scharth: Numerically Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models
Kalman filtering and smoothing (KFS) methods. The second
reinterprets and extends the EIS sampler in Jung, Liesenfeld,
and Richard (2011) as a backward-forward simulation smoothing method for a linear state-space model. We clarify the relations between these two methods. We further conduct a Monte
Carlo and empirical study to analyze the efficiency gain of the
NAIS method when applied to the stochastic volatility model
as, for example, in Ghysels, Harvey, and Renault (1996). We
present results for other model specifications in an online appendix (supplementary materials): the stochastic duration model
of Bauwens and Veredas (2004); the stochastic copula model of
Hafner and Manner (2012); and the dynamic factor model for
multivariate count data of Jung, Liesenfeld, and Richard (2011).
The Monte Carlo study reveals three major findings. First, we
show that our NAIS method can provide 40–50% reductions in
simulation variance for likelihood evaluation, when compared
to a standard implementation of the EIS. Second, the use of
the new control variables further decreases the variance of the
likelihood estimates by 20–35% relative to the use of antithetic
variables as a variance reduction device. The use of antithetic
variance reduction techniques may even lead to a loss of computational efficiency, especially for models with multiple state
variables. Third, by taking the higher computational efficiency
of the NAIS method into account, we find 70–95% gains in variance reduction of the likelihood estimates, when compared to
the standard EIS method and after we have normalized the gains
in computing time. Similar improvements are obtained when we
compare the EIS algorithm of Richard and Zhang (2007) with
the local approximation method of SPDK.
To illustrate the NAIS method in an empirical setting, we
consider a two-factor stochastic volatility model applied to time
series of returns for a set of major U.S. stocks. The two-factor
structure of the volatility specification makes estimation by
means of importance sampling a nontrivial task. However, we
are able to implement the NAIS approach using standard hardware and software facilities without complications. The NAIS
method reduces the computing times in this application by as
much as 66% and leads to Monte Carlo standard errors for the
estimated parameters which are small compared to their statistical standard errors. This application illustrates that we are able
to use the NAIS method effectively for estimation and inference
in many practical situations of interest.
The structure of the article is as follows. Section 2 presents
the nonlinear non-Gaussian state-space model, introduces the
necessary notation, and reviews the key importance sampling
methods. Our main methodological contributions are in Section
2.4 and Section 3, which present our numerically accelerated importance sampling (NAIS) method and the corresponding new
control variables, respectively. Section 4 discusses the results of
the Monte Carlo and empirical studies. Section 5 concludes.
2.
IMPORTANCE SAMPLING FOR STATE-SPACE
MODELS
2.1 Nonlinear and Non-Gaussian State-Space Model
The general ideas of importance sampling are well established and developed in the contributions of Kloek and van
Dijk (1978), Ripley (1987) and Geweke (1989), among others.
Danielsson and Richard (1993), Shephard and Pitt (1997) and
Durbin and Koopman (1997) have explored the implementation
115
of importance sampling methods for the analysis of nonlinear
non-Gaussian time series models. Richard and Zhang (2007)
provided a short review of the literature with additional references. Our main task is to evaluate the likelihood function for
the nonlinear non-Gaussian state-space model
yt |θt ∼ p(yt |θt ; ψ),
αt+1 = dt + Tt αt + ηt ,
θt = Zt αt ,
α1 ∼ N(a1 , P1 ),
t = 1, . . . , n,
ηt ∼ N(0, Qt ),
(1)
where yt is the p × 1 observation vector, θt is the q × 1 signal
vector, αt is the m × 1 state vector, and Zt is the q × m selection matrix; the dynamic properties of the stochastic vectors yt ,
θt , and αt are determined by the m × 1 constant vector dt , the
m × m transition matrix Tt , and m × m variance matrix Qt . The
Gaussian innovation series ηt is serially uncorrelated. The initial mean vector a1 and variance matrix P1 are determined from
the unconditional properties of the state vector αt . The system
variables Zt , dt , Tt , and Qt are only time-varying in a deterministic way. The unknown fixed parameter vector ψ contains, or
is a function of, the unknown coefficients associated with the
observation density p(yt |θt ; ψ) and the system variables.
The nonlinear non-Gaussian state-space model as formulated
in Equation (1) allows the introduction of time-varying parameters in the density p(yt |θt ; ψ). The time-varying parameters
depend on the signal θt in a possibly nonlinear way. The signal
vector θt depends linearly on the state vector αt , for which we
formulate a linear dynamic model. Our general framework accommodates combinations of autoregressive moving average,
long memory, random walk, and cyclical and seasonal dynamic
processes. Harvey (1989) and Durbin and Koopman (2012) provided a detailed discussion of state–space model representations
and unobserved components time series models.
2.2
Likelihood Evaluation via Importance Sampling
Define and y ′ = (y1′ , . . . , yn′ ), θ ′ = (θ1′ , . . . , θn′ ) and α ′ =
, . . . , αn′ ) . If p(yt |θt ; ψ) is a Gaussian density with mean
θt = Zt αt and covariance matrix Ht , for t = 1, . . . , n, Kalman
filtering and smoothing methods evaluate the likelihood and
compute the minimum mean squared error estimates of the state
vector αt together with its mean squared error matrix. In all
other cases, the likelihood for (1) is given by the analytically
intractable integral
L(y; ψ) = p(α, y; ψ) dα
(α1′
=
n
t=1
p(yt |θt ; ψ)p(αt |αt−1 ; ψ) dα1 . . . dαn , (2)
where p(α, y; ψ) is the joint density of y and α following from
(1). Kitagawa (1987) developed a numerical integration method
for evaluating the likelihood integral in (2). This approach is
only feasible when n is small and yt , θt , and αt are scalars.
We aim to evaluate the likelihood function by means of importance sampling. For this purpose, we consider the Gaussian
importance density
g(α|y; ψ) = g(y|α; ψ)g(α; ψ)/g(y; ψ),
116
Journal of Business & Economic Statistics, January 2015
where g(y|θ ; ψ), g(α; ψ) and g(y; ψ) are all Gaussian densities
and where g(y; ψ) can be interpreted as a normalizing constant.
It is implied from (1) that g(α; ψ) ≡ p(α; ψ). We can express
the likelihood function as
p(α, y; ψ)
g(α, y; ψ) dα
L(y; ψ) =
g(α, y; ψ)
p(y|θ ; ψ)p(α; ψ)
g(α|y; ψ) dα
= g(y; ψ)
g(y|θ ; ψ)g(α; ψ)
= g(y; ψ) ω(θ, y; ψ)g(θ |y; ψ) dθ,
where the importance weight function is given by
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
ω(θ, y; ψ) = p(y|θ ; ψ) / g(y|θ ; ψ).
(3)
We evaluate the likelihood function using S independent trajectories θ (1) , . . . , θ (S) that we sample from the signal importance
density g(θ |y; ψ). The likelihood estimate is given by
ψ) = g(y; ψ) × ω,
L(y;
¯
ω¯ =
1
S
S
ωs ,
s=1
ωs = ω(θ (s) , y; ψ),
(4)
where ωs is the importance weight (3) evaluated at θ = θ (s) .
The estimate (4) relies on the typically low-dimensional signal
vector θt rather than the typically high-dimensional state vector
αt . Hence the computations can be implemented efficiently.
Under standard regularity conditions, the weak law of large
numbers ensures that
p
ψ) −→
L(y; ψ),
L(y;
(5)
when S → ∞. A central limit theorem is applicable only when
the variance of the importance weight function exists; see
Geweke (1989). The failure of this condition leads to slow
and unstable convergence of the estimate. Monahan (1993) and
Koopman, Shephard, and Creal (2009) developed diagnostic
tests for validating the existence of the variance of the importance weights based on extreme value theory. Richard and Zhang
(2007) discussed more informal methods for this purpose.
2.3 Importance Density as a Linear State-Space Model
The Gaussian importance density for the state vector can be
represented as g(α|y; ψ) = g(α, y; ψ)/g(y; ψ) with
g(α, y; ψ) =
n
t=1
g(yt |θt ; ψ)g(αt |αt−1 ; ψ),
(6)
where g(αt |αt−1 ; ψ) is the Gaussian density for the state transition equation in (1), and with
1 ′
′
g(yt |θt ; ψ) = exp at + bt θt − θt Ct θt ,
(7)
2
where scalar at , vector bt , and matrix Ct are defined as functions
of the data vector y and the parameter vector ψ, for t = 1, . . . , n.
The constants a1 , . . . , an are chosen such that (6) integrates
to one. The set of unique importance sampling parameters is,
therefore, given by
{b, C} = {b1 , . . . , bn , C1 , . . . , Cn }.
(8)
The state transition density g(αt |αt−1 ; ψ) in (6) represents the
dynamic properties of αt and is the same as in the original model
(1) because g(α; ψ) ≡ p(α; ψ). Hence, the importance density
only varies with {b, C}. We discuss proposals for computing the
importance parameter set {b, C} in Section 2.4.
Koopman, Lit, and Nguyen (2014) show that the observation
density g(yt |θt ; ψ) in (7) can be represented by a linear statespace model for the artificial observations yt∗ = Ct−1 bt which
are computed for a given importance parameter set {b, C} in
(8). The observation equation for yt∗ is given by
yt∗ = θt + εt ,
εt ∼ N(0, Ct−1 ),
t = 1, . . . , n,
(9)
where θt is specified as in (1) and the innovation series εt is
assumed to be serially and mutually uncorrelated with the innovation series ηt in (1). The Gaussian logdensity log g(yt∗ |θt ; ψ)
is equivalent to the log of (7) since
1
1
log g(yt∗ |θt ; ψ) = − log 2π + log |Ct |
2
2
1
− (Ct−1 bt − θt )′ Ct (Ct−1 bt − θt )
2
1
(10)
= at + bt′ θt − θt′ Ct θt ,
2
where the constant at collects all the terms that are not associated
with θt . It follows that at = (log |Ct | − log 2π − bt′ yt∗ )/2. We
conclude that g(θ, y; ψ) ≡ g(θ, y ∗ ; ψ), and hence g(α, y; ψ) ≡
g(α, y ∗ ; ψ), with y ∗ ′ = (y1∗ ′ , . . . , yn∗ ′ ).
The linear model representation allows the use of
computationally efficient methods for signal extraction and simulation smoothing. In particular, for the Monte Carlo evaluation of the likelihood as in (4), we require simulations from
g(θ |y; ψ) ≡ g(θ |y ∗ ; ψ) for a given set {b, C} where b and C
are functions of y and ψ. We have two options that lead to
numerically equivalent results. The first option is to apply the
state-space methods to model (9) as described by Durbin and
Koopman (2012). Signal extraction relies on the Kalman filter
smoother and simulation smoothing may rely on the methods of
de Jong and Shephard (1995) or Durbin and Koopman (2002).
The second option is to use a modification of the importance
sampling method of Jung, Liesenfeld, and Richard (2011) which
we refer to as the backward-forward (BF) method. We show in
Appendix A that the approach of Jung, Liesenfeld, and Richard
(2011) can be reshaped and extended to obtain a new algorithm
for signal extraction and simulation smoothing applied to model
(9). This completes the discussion of likelihood evaluation via
importance sampling. Several devices for simulation variance
reduction, including antithetic variables, can be incorporated
in the computations of (4). Next we discuss a new method for
choosing the importance parameter set {b, C} in (8). Existing
methods are reviewed in Section 2.5.
2.4 Numerically Accelerated Importance Sampling
The numerically accelerated importance sampling (NAIS)
method constructs optimal values for the importance parameter
set {b, C} using the same criterion for the efficient importance
sampling (EIS) method of Liesenfeld and Richard (2003) and
Richard and Zhang (2007). The values for b and C in the EIS
Koopman, Lucas, and Scharth: Numerically Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models
are chosen such that the variance of the log importance weights
log ω(θ, y; ψ) is minimized, that is,
min λ2 (θ, y; ψ)ω(θ, y; ψ)g(θ |y; ψ) dθ,
(11)
b,C
where
λ(θ, y; ψ) = log ω(y, θ ; ψ) = log p(y|θ ; ψ) − log g(y|θ ; ψ),
(12)
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
n
with g(y|θ ; ψ) = t=1 g(yt |θt ; ψ) and with g(yt |θt ; ψ) given
by (7). The minimization in (11) is high-dimensional and numerically not feasible in most cases of interest. The minimization in (11) is therefore reduced to a series of minimizations for
bt and Ct , for t = 1, . . . , n. Hence, we obtain the importance
parameters {bt , Ct } by
min λ2 (θt , yt ; ψ)ω(θt , yt ; ψ)g(θt |y; ψ) dθt , t = 1, . . . , n,
bt ,Ct
(13)
where ω(θt , yt ; ψ) = p(yt |θt ; ψ)/g(yt |θt ; ψ) and λ(θt , yt ; ψ) =
log ω(yt , θt ; ψ).
The NAIS method is based on the insight that the smoothing
density g(θt |y; ψ) ≡ g(θt |y ∗ ; ψ) is available analytically for the
linear Gaussian state-space model (9). For a given set {b, C} and
for a scalar signal θt , we have
1 −1
2
∗
g(θt |y ; ψ) = N(θt , Vt ) = exp − Vt (θt − θt ) / 2π Vt ,
2
117
obtain mean
θt and variance Vt from (14), for t = 1, . . . , n. New
values for bt and Ct are then obtained from the minimization
(15) that is reduced to a weighted least squares computation
θtj ; ψ) as the “dependent” varifor each t. We take log p(yt |
able, vector (1,
θtj , −0.5
θtj2 )′ as the “explanatory” variable and
θtj , yt ; ψ) as the “weight” for which all terms
exp(zj2 /2)h(zj )ω(
that do not depend on zj are dropped. The regression computations are based on sums over j = 1, . . . , M. The second
and third regression estimates, those associated with
θtj and
θtj2 ,
represent the new values for bt and Ct , respectively. The computations are repeated for each t or are done in parallel. Once
a new set for {b, C} is constructed, we can repeat this procedure. Convergence toward a final set {b, C} typically takes a few
iterations.
We start the procedure by having initial values for {b, C}. A
simple choice is bt = 0 and Ct = 1, for t = 1, . . . , n. We can
also initialize {b, C} by the local approximation of Durbin and
Koopman (1997); see Section 2.5. Finally, Richard and Zhang
(2007) argued that we can omit the term ω(
θtj , yt ; ψ) for computing the regression “weight” without much loss of numerical
efficiency. We prefer to adopt this modification because it is
computationally convenient. The NAIS procedure for selecting
the importance parameter set is reviewed in Figure 1.
2.5
Relation to Previous Methods
We show in Section 4 that the NAIS method is an efficient
method from both computational and numerical perspectives.
t = 1, . . . , n,
(14) We compare the NAIS with two related methods. The first is
the local approximation method of Shephard and Pitt (1997)
where the conditional mean
θt and variance Vt are obtained from and Durbin and Koopman (1997) which we refer to as SPDK. It
the Kalman filter and smoother (KFS) or the backward-forward is based on a second-order Taylor expansion of log p(yt |θt ; ψ)
(BF) smoothing method of Appendix A, in both cases applied around the conditional mode of p(θt |y; ψ) which can be computed iteratively using KFS or BF applied to model (9) for
to model (9).
θt and Vt as in the algorithm in Figure 1. The iterThis result allows us to directly minimize the low dimensional evaluating
integral (13) for each time period t via the method of numerical ations require analytic expressions for the first two derivatives
integration. For most cases of practical interest, the method of log p(yt |θt ; ψ), with respect to θt , for t = 1, . . . , n. The conproduces a virtually exact solution. Numerical integration is vergence toward the mode of p(θt |y; ψ) typically takes a few
reviewed in Monahan (2001). We adopt the Gauss–Hermite iterations; see Durbin and Koopman (2012).
The second method is the efficient importance sampling (EIS)
quadrature method that is based on a set of M abscissae zj
with associated weights h(zj ), for j = 1, . . . , M. The value of algorithm as developed by Richard and Zhang (2007). This
M is typically between 20 and 30. The values for the weights method approximates the minimization problem in (13) via the
h(zj ) are predetermined for any value of M. The Gauss–Hermite sampling of the signal θt = Zt αt via simulation smoothing from
quadrature approximation of the minimization (13) is then given g(α|y; ψ), for t = 1, . . . , n, rather than by numerical integration as in NAIS. In the modeling framework (1), the sampling is
by
carried out by the backward-forward sampler of Jung, LiesenM
feld, and Richard (2011). The BF sampler in Appendix A is a
2
min
h(zj ) exp(zj )ϕ(
θtj ),
(15)
modification and a more efficient implementation of this sambt ,Ct
j =1
pler. Furthermore, the introduction of antithetic variables halves
where ϕ(
θtj ) = λ2 (
θtj , yt ; ψ)ω(
θtj , yt ; ψ) g(
θtj |y ∗ ; ψ), with the simulation effort. Koopman, Lit, and Nguyen (2014) de1/2
θtj =
θt + Vt zj , for j = 1, . . . , M. The conditional mean
θt velop an implementation of EIS that is based on the KFS. We
and variance Vt are defined in Equation√(14) from which it fol- emphasize that the EIS uses simulation methods both for selectlows that g(
θtj |y ∗ ; ψ) = exp(−0.5zj2 )/ 2π Vt . Further details ing {b, C} and for evaluating the likelihood function. In Section
of our numerical implementation are provided in the online ap- 4 we consider the methods of SPDK and EIS using BF and
KFS. These methods for selecting the importance parameter set
pendix (supplementary materials).
The minimization (15) takes place iteratively as in the original are reviewed in detail in the online appendix (supplementary
EIS method. For a given importance parameter set {b, C}, we materials).
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
118
Journal of Business & Economic Statistics, January 2015
Figure 1. NAIS algorithm for selecting the importance parameter set {b, C}.
3.
NEW CONTROL VARIABLES FOR NAIS
We introduce a new set of control variables to improve the
numerical efficiency of likelihood evaluation using importance
sampling based on NAIS. We develop control variables that are
based on specific Taylor series expansions. The control variables can be evaluated by numerical integration using Gauss–
Hermite quadrature. We exploit the differences between the
estimates that are obtained from simulation and numerical integration methods with the sole purpose of reducing the variance
of importance sampling estimates. This approach of variance
reduction in the context of NAIS can replace the use of the
antithetic variables proposed by Ripley (1987) and Durbin and
Koopman (2000).
The
likelihood estimate (4) is the sample average ω¯ =
S −1 Ss=1 ωs multiplied by g(y; ψ), where
ωs = ω(θ (s) , y; ψ) =
t = 1, . . . , n,
n
ωts ,
t=1
ωts = ω(θt(s) , yt ; ψ),
s = 1, . . . , S,
with the importance sample weights ω(θt , yt ; ψ) defined below
(13), θ (s) generated from the importance density g(θ |y; ψ), and
θt(s) denoting the tth element of θ (s) , for t = 1, . . . , n. The density g(y; ψ) can be evaluated by the Kalman filter applied to
the linear Gaussian model (9) for some value of χ . The variance of the sample average ω¯ determines the efficiency of the
importance sampling likelihood estimate (4).
To reduce the variance of ω,
¯ we construct control variates
based on
x(θ, y; ψ) = log ω(θ, y; ψ) = log p(y|θ ; ψ) − log g(y|θ ; ψ).
The tth contribution of x(θ, y; ψ) is
given by x(θt , yt ; ψ) =
log ω(θt , yt ; ψ) such that x(θ, y; ψ) = nt=1 x(θt , yt ; ψ). Given
the draws θ (1) , . . . , θ (S) , we define
xs = log(ωs ) =
n
t=1
xts ,
s = 1, . . . , S,
where xts = log(ωts ), and hence ωts = exp(xts ) for t =
1, . . . , n. We can express the sample average of ωs in terms
of xs = log ωs by means of a Taylor series around some value
x, that is,
S
1
1
2
ω¯ = exp(x)
1 + [xs − x] + [xs − x] + · · · .(16)
S s=1
2
We adopt the terms involving xts , t = 1, . . . , n, in this expansion as control variables. Our method consists of replacing the
highest variance terms of the Taylor series by their probability
limits, which we compute efficiently via the NAIS algorithm.
This step clearly leads to a further reduction of the importance
sampling variance and to an improvement of the numerical efficiency at a low computational cost.
3.1 Construction of the First New Control Variable
We base our first control variable on the first order term
(xs − x) of the Taylor series expansion (16). Under the same
regularity conditions required for importance sampling, we have
x¯ =
S
1
p
xs −→
x,
S s=1
(17)
where
x = Eg x(θ, y; ψ) and where Eg is expectation with respect to density g(θ |y; ψ). The Taylor series expansion (16)
Koopman, Lucas, and Scharth: Numerically Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models
around x =
x can now be used to construct a first-order control
variable.
Since
x=
n
Eg [x(θt , yt ; ψ)] ,
t=1
we can evaluate
x by means of the Gauss–Hermite quadrature
method for each time index t separately as discussed in Section
2.4, that is,
xt = Eg [x(θt , yt ; ψ)] = x(θt , yt ; ψ)g(θt |y; ψ) dθt
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
≈
M
j =1
zj2
x(
θtj , yt ; ψ)g(
θtj |y; ψ)h(zj )e ,
1/2
where
θtj =
θt + Vt zj and with the numerical evaluation as
in (15). The Kalman filter and smoother computes
θt and Vt for
xt .
t = 1, . . . , n. Furthermore, we have
x = nt=1
The likelihood estimate (4) corrected for the first control
variable is given by
1
ψ)c = g(y; ψ) exp(
ωs − exp(
x )xs
x )
x+
L(y;
S s
ψ) + g(y; ψ) exp(
¯
= L(y;
x ) (
x − x).
Monte Carlo and empirical evidence of the importance of
the two control variables in reducing the simulation variance
of the likelihood estimate obtained from the NAIS method is
provided in Section 4 for the stochastic volatility model. More
evidence for other models is presented in the online appendix
(supplementary materials).
4.
4.1
MONTE CARLO AND EMPIRICAL EVIDENCE
FOR SV MODEL
Design of Monte Carlo Study
We consider the stochastic volatility (SV) model to illustrate the performance of NAIS in comparison to the alternative
importance sampling methods discussed in Section 2, with or
without the control variables introduced in Section 3. The SV
model may easily be one of the most widely studied nonlinear
state-space models. The references to some key developments
in the SV model literature are Tauchen and Pitts (1983), Taylor
(1986), and Melino and Turnbull (1990). Ghysels, Harvey, and
Renault (1996) and Shephard (2005) provided detailed reviews
of SV models.
For a time series of log-returns yt , we consider the k-factor
univariate stochastic volatility model of Liesenfeld and Richard
(2003) and Durham (2006), that is given by
yt ∼ N(0, σt2 ),
p
ψ)c −→ L(y; ψ). When
It follows from (5) and (17) that L(y;
the importance model (9) provides an accurate approximation
to the likelihood, ωs is close to one and xs is close to zero, such
that ωs ≈ 1 + xs . Hence, ωs and exp(
x )xs are typically highly
and positively correlated. When the importance model is a less
accurate approximation, the positive correlation remains, but at
ψ)c is a more efficient
a more moderate level. Therefore, L(y;
ψ).
estimate of the likelihood compared to L(y;
3.2 Construction of the Second New Control Variable
We base our second control variable on the second-order
term (xs − x)2 of the Taylor series expansion (16). We aim to
correct for the sample variation of (xts −
xt )2 within the sample
(1)
(S)
xt is the tth
of draws θt , . . . , θt for each t individually, where
element of
x . Using the same arguments as in Section 3.1, we
p
σt2 , where
have σ¯ t2 −→
σ¯ t2 =
σt2
1
(xts −
xt )2 ,
S
2
= Eg (xts −
xt ) =
(xts −
xt )2 g(θt |y; ψ) dθt .
We compute the variance
σt2 using the Gauss–Hermite quadrature. Define
n
ψ)cc = L(y;
ψ)c + 1 g(y; ψ) exp(
x)
(
σt2 − σ¯ t2 ),
L(y;
2
t=1
p
ψ)cc −→
L(y; ψ). Since we
from which it follows that L(y;
xt )2 by its probability
can replace the sample variation of (xts −
ψ)cc to be more efficient than
limit, we can expect estimate L(y;
ψ) and L(y;
ψ)c .
L(y;
119
σt2 = exp(c + θt ),
θt = α1,t + · · · + αk,t ,
t = 1, . . . , n,
with scalar constant c, scalar signal θt and k × 1 state vector
αt = (α1,t , . . . , αk,t )′ . The state vector is modeled by (1) with
k × k diagonal time-invariant matrices Tt and Qt given by
⎤
⎡ 2
⎡
⎤
ση,1 0
0
0
φ1 0
⎥
⎥
⎢
⎢
..
..
⎥
Qt = ⎢
Tt = ⎢
. 0⎥
.
0 ⎦,
⎦,
⎣ 0
⎣0
0
0
φk
0
0
2
ση,k
2
and with unknown coefficients |φi | < 1 and ση,i
> 0, for i =
1, . . . , k. We identify the parameters by imposing φ1 > · · · >
φk . The signal θt represents the log-volatility.
For the purpose of likelihood evaluation by importance sampling, we examine the performance of the SPDK method, two
implementations of the standard EIS method and four implementations of our NAIS method. The SPDK method is based on
the mode approximation that is computed iteratively using the
Kalman filter and smoother (KFS); see Section 2.5. The likelihood function is evaluated using the simulation smoother of
de Jong and Shephard (1995) or Durbin and Koopman (2002)
which we refer to as JSDK. The EIS method is based on a Monte
Carlo approximation to obtain the importance parameter set via
the minimization (13); see Section 2.5. Our proposed NAIS
method is introduced in Section 2.4. The likelihood evaluation
using EIS and NAIS can both be based on the JSDK simulation
smoother or the BF sampler of Appendix A. Finally, both NAIS
implementations can be extended with the use of the new control
variables introduced in Section 3. Table 1 reviews the methods
and their different implementations.
The design of the Monte Carlo study is as follows. We consider 500 random time series for each of the two SV models in
120
Journal of Business & Economic Statistics, January 2015
Table 1. Importance sampling methods for state-space models
I
Task
SPDK
EIS-BF
EIS-JSDK
IS parameter set {b, C} via
constructed by simulation
or by numerical integration
using method
Local mode
—
—
KFS
(13)
√
(13)
√
—
BF
—
JSDK
JSDK
BF
JSDK
—
—
—
II
Simulation method
III
Control variables (optional)
NAIS-BF
(-Ctrl)
NAIS-JSDK
(-Ctrl)
(13)
—
√
(13)
—
√
BF
KFS
BF
√
( )
JSDK
√
( )
NOTE: Likelihood evalution via importance sampling: I, obtain importance sampling (IS) parameter set {b, c} using methods described in Section 2.4 and 2.5; II, Sampling from the
importance density to construct likelihood function; III, in case of NIAS, with use of control variable in Section 3 or not.
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
our study. The first SV model has a single log-volatility component, k = 1, with coefficients
c = 1,
φ1 = 0.98,
2
ση,1
= 0.0225,
which are typical values in empirical studies for daily stock
returns. The second SV model has k = 2 with
c = 1,
φ2 = 0.9,
φ1 = 0.99,
2
ση,1
= 0.005,
2
ση,2
= 0.03.
We take 500 simulated time series to avoid the dependence of
our conclusions on particular trajectories of the simulated states
and series. For each simulated time series, we estimate the loglikelihood function at the true parameters a hundred times using
different common random numbers. Hence, each cell in the
tables presented below reflects 50,000 likelihood evaluations.
We report the results for two different sample sizes, n = 1000
and n = 5000, and we use S = 200 importance samples for each
likelihood evaluation and for each method. The number of nodes
for the numerical integration calculations is set to M = 20.
We start by estimating the variance and the bias associated
with each importance sampling method. We compute the reported statistics as
Bias =
Variance =
500
100
1
j (y i ; ψ) − log L(y i ; ψ) ,
·
log L
50, 000 i=1 j =1
500
100
1 1
j (y i ; ψ)
·
·
log L
500 i=1 100 j =1
2
− log L(y i ; ψ) ,
(18)
where y i is the ith simulated time series, log L(y i ; ψ) is the
j (y i ; ψ) is the estimate of
“true” log-likelihood value, log L
the log-likelihood function for a particular method and for
the jth set of common random
numbers, j = 1, . . . , 100,
log
Lj (y i ; ψ). The true logand log L(y i ; ψ) = 100−1 100
j =1
likelihood value is unknown but we take its “true” value as
the log of the average of likelihood estimates from the NAIS
method with S = 200 × 100 = 20,000 importance samples.
We expect the approximation error with respect to the true
likelihood to be small. We compute the mean squared error
(MSE) as the sum of the variance and the squared bias. The
variance and the MSE are reported as a ratios with respect to
the EIS-BF method; see Table 1.
The numerical efficiency of estimation via simulation can be
increased by generating additional samples. A systematic comparison between methods must therefore take into account computational efficiency. We report the average computing times
for each method on the basis of an Intel Duo Core 2.5GHz
processor. We record the times required for constructing the importance sampling parameter set (task I) and for jointly generating importance samples and computing the likelihood estimate
(task II). In case of NAIS with control variables, we include
the additional time for task III in the total time of task II. Our
key summary statistic is the time normalized variance ratio of
method a against the benchmark method b and it is given by
−1
TimeaI+II − TimebI+II
,
(19)
Variancea/b × 1 +
TimeaII
where Variancea/b is the ratio of the variance defined in (18)
for methods a and b and Timem
j is the time length of task j, for
j = I, II, I+II, by method m, for m = a, b. We have excluded
TimeaI from the denominator because it is a fixed cost and not
relevant for drawing additional samples.
To have the computing times of EIS and NAIS implementations comparable to each other, we initialize the minimization of
(11) by the local approximation for {b, C} of the SPDK method.
To reduce the simulation variance for all methods, we use antithetic variables for location as in Durbin and Koopman (2000)
except for the NAIS methods that include the control variables
of Section 3. We have found no evidence of importance sampling weights that constitute an infinite variance in our study;
see the discussions in Koopman, Shephard, and Creal (2009).
Our diagnostic procedure includes the verification of how sensitive the importance sampling weights are to artificial outliers as
in Richard and Zhang (2007). We have efficiently implemented
all methods using MATLAB and C.
4.2 Monte Carlo Results: Log-Likelihood Estimation
Table 2 presents the results for the stochastic volatility model
with k = 1. The results show that our numerical integration
method for constructing the importance density leads to 40%–
50% reductions in the variance of the log-likelihood estimates
compared to the EIS method. In all cases the numerical gains
become larger when we increase the time series dimension from
n = 1000 to n = 5000. The use of the control variable further
increases the efficiency of the NAIS method as it generates a
further 20% –35% gain in producing a smaller variance. We find
no deterioration in relative performance for the MSE measure,
Koopman, Lucas, and Scharth: Numerically Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models
121
Table 2. Log-likelihood errors for stochastic volatility model, k = 1
n = 1000, S = 200
SPDK
EIS-BF
EIS-JSDK
NAIS-BF
NAIS-JSDK
NAIS-BF-Ctrl
NAIS-JSDK-Ctrl
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
n = 5000, S = 200
SPDK
EIS-BF
EIS-JSDK
NAIS-BF
NAIS-JSDK
NAIS-BF-Ctrl
NAIS-JSDK-Ctrl
Variance
MSE
Time step 1
(×10)
Time step 2
(×10)
TNVAR
12.779
1.000
1.009
0.595
0.594
0.405
0.415
12.837
1.000
1.009
0.595
0.594
0.406
0.416
0.022
0.232
0.247
0.073
0.070
0.073
0.073
0.173
0.187
0.172
0.174
0.175
0.192
0.180
5.576
1.000
1.005
0.299
0.297
0.224
0.216
Variance
MSE
Time step 1
(×10)
Time step 2
(×10)
TNVAR
15.025
1.000
0.997
0.503
0.501
0.380
0.375
18.594
1.000
0.998
0.501
0.499
0.381
0.376
0.052
1.278
1.246
0.364
0.340
0.418
0.398
0.908
0.990
0.908
0.928
0.908
1.002
0.939
6.152
1.000
0.885
0.245
0.236
0.206
0.189
NOTE: The table presents the numerical and computational performance of different IS methods for log-likelihood estimation. We simulate 500 different realizations from the model.
For each of these realizations, we obtain log-likelihood estimates for 100 different sets of random numbers. We estimate the variance associated with each method as the average sample
variance across the 500 realizations. We define the mean-square error (MSE) as the sum of the variance and the square of the average bias across the 500 realizations. We show these
statistics as ratios with the standard implementation of the EIS-BF method as the benchmark. The time for step 1 column gives the fixed time cost for obtaining the parameters of the
importance density, while the time for step 2 refers to the computational cost of sampling from the importance density and calculating the likelihood estimate. The TNVAR column reports
the time normalized variance ratio according to (19). Table 1 lists the methods used and their acronyms. NAIS-BF-Ctrl and NAIS-JSDK-Ctrl refer to the NAIS methods with use of the
control variables of Section 3. We specify the stochastic volatility model as: yt ∼ N(0, σt2 ) with σt2 = exp(αt ) and αt+1 = 0.98αt + ηt where ηt ∼ N(0, ση2 = 0.0225) for t = 1, . . . , n.
confirming the accuracy of the numerical integration method for
obtaining the control variables.
The results further show that the NAIS method can also
achieve substantial gains in computational efficiency. It is able
to construct the importance density 70% faster than the EIS
method. This result is partly due to the ability of working with
the marginal densities and the scope for optimizing the computer
code in this setting. In all cases the EIS method takes longer to
construct the importance density than it does to generate the
importance samples and to compute the log-likelihood estimate
based on S = 200 simulation samples. When we normalize the
variances by the length of computing time, we also obtain gains
for the NAIS method. Table 2 presents a total improvement of
70%–80% by the NAIS method with control variables. The gain
in performance is comparable when the default EIS method is
compared with the SPDK method. The results further indicate
that the computing times for the BF and JSDK simulation methods are equivalent for the SV model with k = 1.
Table 3 reports the findings for the SV model with two logvolatility components, k = 2. These results are overall similar
Table 3. Log-likelihood errors for stochastic volatility model, k = 2
n = 1000, S = 200
SPDK
EIS-BF
EIS-JSDK
NAIS-BF
NAIS-JSDK
NAIS-BF-Ctrl
NAIS-JSDK-Ctrl
n = 5000, S = 200
SPDK
EIS-BF
EIS-JSDK
NAIS-BF
NAIS-JSDK
NAIS-BF-Ctrl
NAIS-JSDK-Ctrl
Variance
MSE
Time step 1
(×10)
Time step 2
(×10)
TNVAR
17.523
1.000
1.011
0.562
0.562
0.374
0.374
17.539
1.000
1.001
0.564
0.554
0.378
0.373
0.223
0.763
0.578
0.569
0.364
0.592
0.385
0.238
0.374
0.314
0.304
0.237
0.408
0.307
4.572
1.000
0.569
0.301
0.172
0.280
0.153
Variance
MSE
Time step 1
(×10)
Time step 2
(×10)
TNVAR
18.915
1.000
0.972
0.483
0.474
0.368
0.365
26.283
1.000
0.963
0.484
0.465
0.371
0.363
1.341
3.988
3.067
3.010
1.976
3.089
2.050
1.293
1.962
1.673
1.619
1.298
2.152
1.655
5.305
1.000
0.564
0.266
0.155
0.277
0.155
NOTE: We refer to the description of Table 2. We specify the two-factor stochastic volatility model as: yt ∼ N(0, σt2 ) with σt2 = exp(αt,1 + αt,2 ), αt+1,1 = 0.99αt,1 + ηt,1 , αt+1,1 =
2
2
0.9αt,2 + ηt,2 , where ηt,1 ∼ N(0, ση,1
= 0.005), ηt,2 ∼ N(0, ση,2
= 0.03), for t = 1, . . . , n.
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
122
Journal of Business & Economic Statistics, January 2015
to the ones for the SV model with k = 1 in Table 2. However, we
find two important differences. First, for a model with multiple
states, as in the SV model with k = 2, we achieve a small gain
in computational efficiency by switching from the BF sampler
to the JSDK sampler. The main reason is that JSDK is able to
simulate the univariate signal θt directly whereas the BF sampler
needs to simulate the signal via the (multiple) state vector αt .
Hence, we obtain around 83% reductions in the variance normalized by time when using the NAIS-JSDK method instead of
the EIS-BF method.
The second difference between the results for the SV model
with k = 1 and k = 2 concerns the variance reduction that we
obtain by using control variables in the likelihood estimation.
For the case of k = 2, the variance reductions due to the
control variables are similar to those for k = 1, but the relative
increase in the computational cost for step 2 is higher. This is
because antithetic variables reduce the computational effort for
simulation.
4.3 Parameter Estimation for the SV Model Using NAIS
To illustrate the performance of our proposed NAIS method
in more detail, we consider the simulated maximum likelihood
estimation of the unknown parameters in a multifactor stochastic
volatility model. We report our findings from both a Monte Carlo
study and an empirical study below. We carry out parameter
estimation using the following consecutive steps:
1. Set a starting value for the estimate of parameter vector ψ.
2. Set starting values for the importance sampling parameter χ
in (8).
3. Maximize the log-likelihood function with respect to ψ using
the NAIS-JSDK method with S = 0 and only the first control
variate; use the estimates as starting values for the next step.
4. Maximize the log-likelihood function with respect to ψ using
the NAIS-JSDK-Ctrl method with S > 0.
The estimation in Step 3 concerns an approximate loglikelihood function. It is fast and requires no simulation, only
numerical integration. The computational efficiency of this procedure is primarily due to the accurate approximation of the
log-likelihood function calculated by the NAIS-JSDK method
with S = 0 and the first control variate from Section 3. As a result, the convergence of the maximization in the last step is fast
as it only requires a small number of iterations. The consequence
is that we can set S at a high value in Step 4 as it only marginally
increases the required computing time. The maximizations in
Steps 3 and 4 are based on a quasi-Newton method. The use
of common random numbers in Step 4 for each likelihood
estimation leads to a smooth likelihood function in ψ, which
is necessary for the application of the quasi-Newton optimization methods. In both studies below we set S = 200 in
Step 4.
4.3.1 Monte Carlo Evidence. We consider the k-factor
stochastic volatility model with k = 3 for time series of lengths
n = 5000 and n = 10,000. The true parameter values are
2
= 0.005,
set to d = 0.5, φ1 = 0.99, φ2 = 0.9, φ3 = 0.4, ση,1
2
2
ση,2 = 0.016, and ση,3 = 0.05. We take these parameter values
also as the starting values for parameter estimation. We draw 100
Table 4. Parameter estimation for stochastic volatility model, k = 3
n = 5000
c
φ1
2
σ1,η
φ2
2
σ2,η
φ3
2
σ3,η
True
Mean
MC
Std. error
Total
Std. error
MC variance
Ratio
0.500
0.990
0.005
0.900
0.015
0.400
0.050
0.489
0.989
0.005
0.883
0.014
0.314
0.054
0.003
0.000
0.000
0.009
0.001
0.025
0.003
0.098
0.005
0.002
0.055
0.007
0.224
0.032
0.001
0.010
0.007
0.029
0.009
0.012
0.006
n = 10,000
c
φ1
2
σ1,η
φ2
2
σ2,η
φ3
2
σ3,η
True
Mean
MC
Std. Error
Total
Std. Error
MC variance
Ratio
0.500
0.990
0.005
0.900
0.015
0.400
0.050
0.500
0.990
0.005
0.893
0.015
0.334
0.054
0.003
0.000
0.000
0.005
0.001
0.061
0.004
0.078
0.003
0.002
0.034
0.005
0.205
0.023
0.001
0.008
0.010
0.021
0.046
0.090
0.035
NOTE: We simulate 100 trajectories of a three-factor stochastic volatility model. For each of these realizations, we obtain 20 simulated maximum likelihood parameter estimates based
on different sets of common random numbers and using the NAIS-JSDK method. We first show the average parameter estimates across the 500 replications. The Monte Carlo (MC)
standard error column reports the square root of the average of the sample variance of the parameter estimates across the 100 realizations. The total standard error column shows the
square-root of the sum of the MC variance and the variance of the average estimates across the 100 trajectories. Finally, we obtain the MC variance ratio by dividing the MC variance by
the total variance. The starting values are the true parameters. The average computing time was 65s and 123s for n = 5000 and n = 10,000, respectively. yt ∼ N(0, σt2 ), t = 1, . . . , n,
σt2 = exp(θt ), θt = d + α1,t + α2,t + α3,t , αt = T αt−1 + ηt , α1 ∼ N(a1 , P1 ), ηt ∼ N(0, Q), where T is a diagonal matrix with elements φ1 , φ2 , and φ3 and Q is a diagonal matrix with
2
2
2
elements ση,1
, ση,2
, and ση,3
.
Koopman, Lucas, and Scharth: Numerically Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models
123
Table 5. Two-factor stochastic volatility model: Empirical study
GE
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
NAIS
JP Morgan
EIS
NAIS
EIS
Coca-Cola
NAIS
EIS
AT&T
NAIS
EIS
Wal-Mart
NAIS
EIS
Exxon
NAIS
EIS
c
0.845
0.845
[0.002] [0.006]
(0.690)
1.275
1.274
[0.003] [0.005]
(0.458)
0.004
0.005
[0.004] [0.007]
(0.393)
0.655
0.655
[0.001] [0.002]
(0.652)
0.375
0.374
[0.002] [0.003]
(0.413)
0.555
0.555
[0.000] [0.001]
(0.193)
φ1
0.997
0.997
[0.000] [0.000]
(0.002)
0.997
0.997
[0.000] [0.000]
(0.002)
0.995
0.995
[0.000] [0.001]
(0.003)
0.997
0.997
[0.000] [0.000]
(0.002)
0.996
0.996
[0.000] [0.000]
(0.002)
0.991
0.991
[0.000] [0.000]
(0.006)
2
σ1,η
0.007
0.007
[0.000] [0.000]
(0.003)
0.008
0.008
[0.000] [0.000]
(0.003)
0.006
0.006
[0.000] [0.001]
(0.003)
0.005
0.005
[0.000] [0.000]
(0.003)
0.005
0.005
[0.000] [0.000]
(0.002)
0.008
0.008
[0.000] [0.001]
(0.008)
φ2
0.458
0.455
[0.016] [0.042]
(0.271)
0.839
0.840
[0.004] [0.006]
(0.065)
0.488
0.484
[0.037] [0.076]
(0.164)
0.897
0.898
[0.002] [0.004]
(0.081)
0.275
0.274
[0.013] [0.028]
(0.118)
0.936
0.936
[0.000] [0.002]
(0.048)
2
σ2,η
0.261
0.261
[0.008] [0.017]
(0.091)
0.086
0.085
[0.002] [0.003]
(0.032)
0.307
0.305
[0.017] [0.029]
(0.077)
0.027
0.027
[0.000] [0.001]
(0.018)
0.263
0.262
[0.007] [0.009]
(0.054)
0.026
0.026
[0.000] [0.001]
(0.013)
86
–65%
67
–66%
69
–52%
74
–65%
36
–51%
46
–47%
Time (seconds)
Improvement
244
196
146
212
73
86
NOTE: We estimate a two-component stochastic volatility specification for the daily returns of six Dow Jones index stocks in the period between January 2001 and December 2010 (2512
observations). We repeat the NAIS-JSDK and EIS-BF estimation methods a hundred times with different random numbers. The Monte Carlo and statistical standard errors are in brackets
[ ] and parenthese
ISSN: 0735-0015 (Print) 1537-2707 (Online) Journal homepage: http://www.tandfonline.com/loi/ubes20
Numerically Accelerated Importance Sampling for
Nonlinear Non-Gaussian State-Space Models
Siem Jan Koopman, André Lucas & Marcel Scharth
To cite this article: Siem Jan Koopman, André Lucas & Marcel Scharth (2015) Numerically
Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models, Journal of
Business & Economic Statistics, 33:1, 114-127, DOI: 10.1080/07350015.2014.925807
To link to this article: http://dx.doi.org/10.1080/07350015.2014.925807
View supplementary material
Accepted author version posted online: 12
Jun 2014.
Submit your article to this journal
Article views: 242
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=ubes20
Download by: [Universitas Maritim Raja Ali Haji]
Date: 11 January 2016, At: 19:19
Supplementary materials for this article are available online. Please go to http://tandfonline.com/r/JBES
Numerically Accelerated Importance Sampling
for Nonlinear Non-Gaussian State-Space
Models
Siem Jan KOOPMAN
Department of Econometrics, VU University Amsterdam, 1081 HV Amsterdam, The Netherlands;
Tinbergen Institute Amsterdam, The Netherlands; CREATES, Aarhus University, DK-8210 Aarhus, Denmark
(s.j.koopman@vu.nl)
Andre´ LUCAS
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
Department of Finance, VU University Amsterdam, 1081 HV Amsterdam, The Netherlands; Tinbergen Institute
and Duisenberg School of Finance, 1082 MS Amsterdam, The Netherlands (a.lucas@vu.nl)
Marcel SCHARTH
Australian School of Business, University of New South Wales, Kensington NSW 2052, Australia
(m.scharth@unsw.edu.au)
We propose a general likelihood evaluation method for nonlinear non-Gaussian state-space models using
the simulation-based method of efficient importance sampling. We minimize the simulation effort by
replacing some key steps of the likelihood estimation procedure by numerical integration. We refer to this
method as numerically accelerated importance sampling. We show that the likelihood function for models
with a high-dimensional state vector and a low-dimensional signal can be evaluated more efficiently using
the new method. We report many efficiency gains in an extensive Monte Carlo study as well as in an
empirical application using a stochastic volatility model for U.S. stock returns with multiple volatility
factors. Supplementary materials for this article are available online.
KEY WORDS: Control variables; Efficient importance sampling; Kalman filter; Numerical integration;
Simulated maximum likelihood; Simulation smoothing; Stochastic volatility model.
1.
INTRODUCTION
The evaluation of an analytically intractable likelihood function is a challenging task for many statistical and econometric
time series models. The key challenge is the computation of
a high-dimensional integral which is typically carried out by
importance sampling methods. Advances in importance sampling over the past three decades have contributed to the interest in nonlinear non-Gaussian state-space models that in
most cases lack a tractable likelihood expression. Examples
include stochastic volatility models as in Ghysels, Harvey, and
Renault (1996), stochastic conditional intensity models as in
Bauwens and Hautsch (2006), non-Gaussian unobserved components time series models as in Durbin and Koopman (2000),
and flexible nonlinear panel data models with unobserved heterogeneity as in Heiss (2008).
We propose a new numerically and computationally efficient
importance sampling method for nonlinear non-Gaussian statespace models. We show that a major part of the likelihood
evaluation procedure can be done by fast numerical integration rather than Monte Carlo integration only. Our main contribution consists of two parts. First, a numerical integration
scheme is developed to construct an efficient importance density that minimizes the log-variance of the simulation error. Our
approach is based on the efficient importance sampling (EIS)
method of Richard and Zhang (2007), which relies on Monte
Carlo simulations to obtain the importance density. Second, we
propose new control variables that eliminate the first order simulation error in evaluating the likelihood function via importance
sampling.
Numerical integration is generally highly accurate but its
feasibility is limited to low-dimensional problems. Although
the Monte Carlo integration method is applicable to high dimensional problems, it is subject to simulation error. These
properties are typical to applications in time series modeling.
Here we adopt both methods and we show how to carry the
virtues of numerical integration over to high dimensional statespace models. We depart from the numerical approaches of
Kitagawa (1987) and Fridman and Harris (1998) as well as from
the simulation based methods of Danielsson and Richard (1993)
and Durbin and Koopman (1997). We refer to our new method
as numerically accelerated importance sampling (NAIS).
Following Shephard and Pitt (1997) and Durbin and Koopman
(1997) (referred to as SPDK), we base our importance sampling
method on an approximating linear Gaussian state-space model
and use computationally efficient methods for this class of models. We explore two different, but numerically equivalent, sets
of algorithms. The first approach follows SPDK and considers
114
© 2015 American Statistical Association
Journal of Business & Economic Statistics
January 2015, Vol. 33, No. 1
DOI: 10.1080/07350015.2014.925807
Color versions of one or more of the figures in the article can be
found online at www.tandfonline.com/r/jbes.
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
Koopman, Lucas, and Scharth: Numerically Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models
Kalman filtering and smoothing (KFS) methods. The second
reinterprets and extends the EIS sampler in Jung, Liesenfeld,
and Richard (2011) as a backward-forward simulation smoothing method for a linear state-space model. We clarify the relations between these two methods. We further conduct a Monte
Carlo and empirical study to analyze the efficiency gain of the
NAIS method when applied to the stochastic volatility model
as, for example, in Ghysels, Harvey, and Renault (1996). We
present results for other model specifications in an online appendix (supplementary materials): the stochastic duration model
of Bauwens and Veredas (2004); the stochastic copula model of
Hafner and Manner (2012); and the dynamic factor model for
multivariate count data of Jung, Liesenfeld, and Richard (2011).
The Monte Carlo study reveals three major findings. First, we
show that our NAIS method can provide 40–50% reductions in
simulation variance for likelihood evaluation, when compared
to a standard implementation of the EIS. Second, the use of
the new control variables further decreases the variance of the
likelihood estimates by 20–35% relative to the use of antithetic
variables as a variance reduction device. The use of antithetic
variance reduction techniques may even lead to a loss of computational efficiency, especially for models with multiple state
variables. Third, by taking the higher computational efficiency
of the NAIS method into account, we find 70–95% gains in variance reduction of the likelihood estimates, when compared to
the standard EIS method and after we have normalized the gains
in computing time. Similar improvements are obtained when we
compare the EIS algorithm of Richard and Zhang (2007) with
the local approximation method of SPDK.
To illustrate the NAIS method in an empirical setting, we
consider a two-factor stochastic volatility model applied to time
series of returns for a set of major U.S. stocks. The two-factor
structure of the volatility specification makes estimation by
means of importance sampling a nontrivial task. However, we
are able to implement the NAIS approach using standard hardware and software facilities without complications. The NAIS
method reduces the computing times in this application by as
much as 66% and leads to Monte Carlo standard errors for the
estimated parameters which are small compared to their statistical standard errors. This application illustrates that we are able
to use the NAIS method effectively for estimation and inference
in many practical situations of interest.
The structure of the article is as follows. Section 2 presents
the nonlinear non-Gaussian state-space model, introduces the
necessary notation, and reviews the key importance sampling
methods. Our main methodological contributions are in Section
2.4 and Section 3, which present our numerically accelerated importance sampling (NAIS) method and the corresponding new
control variables, respectively. Section 4 discusses the results of
the Monte Carlo and empirical studies. Section 5 concludes.
2.
IMPORTANCE SAMPLING FOR STATE-SPACE
MODELS
2.1 Nonlinear and Non-Gaussian State-Space Model
The general ideas of importance sampling are well established and developed in the contributions of Kloek and van
Dijk (1978), Ripley (1987) and Geweke (1989), among others.
Danielsson and Richard (1993), Shephard and Pitt (1997) and
Durbin and Koopman (1997) have explored the implementation
115
of importance sampling methods for the analysis of nonlinear
non-Gaussian time series models. Richard and Zhang (2007)
provided a short review of the literature with additional references. Our main task is to evaluate the likelihood function for
the nonlinear non-Gaussian state-space model
yt |θt ∼ p(yt |θt ; ψ),
αt+1 = dt + Tt αt + ηt ,
θt = Zt αt ,
α1 ∼ N(a1 , P1 ),
t = 1, . . . , n,
ηt ∼ N(0, Qt ),
(1)
where yt is the p × 1 observation vector, θt is the q × 1 signal
vector, αt is the m × 1 state vector, and Zt is the q × m selection matrix; the dynamic properties of the stochastic vectors yt ,
θt , and αt are determined by the m × 1 constant vector dt , the
m × m transition matrix Tt , and m × m variance matrix Qt . The
Gaussian innovation series ηt is serially uncorrelated. The initial mean vector a1 and variance matrix P1 are determined from
the unconditional properties of the state vector αt . The system
variables Zt , dt , Tt , and Qt are only time-varying in a deterministic way. The unknown fixed parameter vector ψ contains, or
is a function of, the unknown coefficients associated with the
observation density p(yt |θt ; ψ) and the system variables.
The nonlinear non-Gaussian state-space model as formulated
in Equation (1) allows the introduction of time-varying parameters in the density p(yt |θt ; ψ). The time-varying parameters
depend on the signal θt in a possibly nonlinear way. The signal
vector θt depends linearly on the state vector αt , for which we
formulate a linear dynamic model. Our general framework accommodates combinations of autoregressive moving average,
long memory, random walk, and cyclical and seasonal dynamic
processes. Harvey (1989) and Durbin and Koopman (2012) provided a detailed discussion of state–space model representations
and unobserved components time series models.
2.2
Likelihood Evaluation via Importance Sampling
Define and y ′ = (y1′ , . . . , yn′ ), θ ′ = (θ1′ , . . . , θn′ ) and α ′ =
, . . . , αn′ ) . If p(yt |θt ; ψ) is a Gaussian density with mean
θt = Zt αt and covariance matrix Ht , for t = 1, . . . , n, Kalman
filtering and smoothing methods evaluate the likelihood and
compute the minimum mean squared error estimates of the state
vector αt together with its mean squared error matrix. In all
other cases, the likelihood for (1) is given by the analytically
intractable integral
L(y; ψ) = p(α, y; ψ) dα
(α1′
=
n
t=1
p(yt |θt ; ψ)p(αt |αt−1 ; ψ) dα1 . . . dαn , (2)
where p(α, y; ψ) is the joint density of y and α following from
(1). Kitagawa (1987) developed a numerical integration method
for evaluating the likelihood integral in (2). This approach is
only feasible when n is small and yt , θt , and αt are scalars.
We aim to evaluate the likelihood function by means of importance sampling. For this purpose, we consider the Gaussian
importance density
g(α|y; ψ) = g(y|α; ψ)g(α; ψ)/g(y; ψ),
116
Journal of Business & Economic Statistics, January 2015
where g(y|θ ; ψ), g(α; ψ) and g(y; ψ) are all Gaussian densities
and where g(y; ψ) can be interpreted as a normalizing constant.
It is implied from (1) that g(α; ψ) ≡ p(α; ψ). We can express
the likelihood function as
p(α, y; ψ)
g(α, y; ψ) dα
L(y; ψ) =
g(α, y; ψ)
p(y|θ ; ψ)p(α; ψ)
g(α|y; ψ) dα
= g(y; ψ)
g(y|θ ; ψ)g(α; ψ)
= g(y; ψ) ω(θ, y; ψ)g(θ |y; ψ) dθ,
where the importance weight function is given by
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
ω(θ, y; ψ) = p(y|θ ; ψ) / g(y|θ ; ψ).
(3)
We evaluate the likelihood function using S independent trajectories θ (1) , . . . , θ (S) that we sample from the signal importance
density g(θ |y; ψ). The likelihood estimate is given by
ψ) = g(y; ψ) × ω,
L(y;
¯
ω¯ =
1
S
S
ωs ,
s=1
ωs = ω(θ (s) , y; ψ),
(4)
where ωs is the importance weight (3) evaluated at θ = θ (s) .
The estimate (4) relies on the typically low-dimensional signal
vector θt rather than the typically high-dimensional state vector
αt . Hence the computations can be implemented efficiently.
Under standard regularity conditions, the weak law of large
numbers ensures that
p
ψ) −→
L(y; ψ),
L(y;
(5)
when S → ∞. A central limit theorem is applicable only when
the variance of the importance weight function exists; see
Geweke (1989). The failure of this condition leads to slow
and unstable convergence of the estimate. Monahan (1993) and
Koopman, Shephard, and Creal (2009) developed diagnostic
tests for validating the existence of the variance of the importance weights based on extreme value theory. Richard and Zhang
(2007) discussed more informal methods for this purpose.
2.3 Importance Density as a Linear State-Space Model
The Gaussian importance density for the state vector can be
represented as g(α|y; ψ) = g(α, y; ψ)/g(y; ψ) with
g(α, y; ψ) =
n
t=1
g(yt |θt ; ψ)g(αt |αt−1 ; ψ),
(6)
where g(αt |αt−1 ; ψ) is the Gaussian density for the state transition equation in (1), and with
1 ′
′
g(yt |θt ; ψ) = exp at + bt θt − θt Ct θt ,
(7)
2
where scalar at , vector bt , and matrix Ct are defined as functions
of the data vector y and the parameter vector ψ, for t = 1, . . . , n.
The constants a1 , . . . , an are chosen such that (6) integrates
to one. The set of unique importance sampling parameters is,
therefore, given by
{b, C} = {b1 , . . . , bn , C1 , . . . , Cn }.
(8)
The state transition density g(αt |αt−1 ; ψ) in (6) represents the
dynamic properties of αt and is the same as in the original model
(1) because g(α; ψ) ≡ p(α; ψ). Hence, the importance density
only varies with {b, C}. We discuss proposals for computing the
importance parameter set {b, C} in Section 2.4.
Koopman, Lit, and Nguyen (2014) show that the observation
density g(yt |θt ; ψ) in (7) can be represented by a linear statespace model for the artificial observations yt∗ = Ct−1 bt which
are computed for a given importance parameter set {b, C} in
(8). The observation equation for yt∗ is given by
yt∗ = θt + εt ,
εt ∼ N(0, Ct−1 ),
t = 1, . . . , n,
(9)
where θt is specified as in (1) and the innovation series εt is
assumed to be serially and mutually uncorrelated with the innovation series ηt in (1). The Gaussian logdensity log g(yt∗ |θt ; ψ)
is equivalent to the log of (7) since
1
1
log g(yt∗ |θt ; ψ) = − log 2π + log |Ct |
2
2
1
− (Ct−1 bt − θt )′ Ct (Ct−1 bt − θt )
2
1
(10)
= at + bt′ θt − θt′ Ct θt ,
2
where the constant at collects all the terms that are not associated
with θt . It follows that at = (log |Ct | − log 2π − bt′ yt∗ )/2. We
conclude that g(θ, y; ψ) ≡ g(θ, y ∗ ; ψ), and hence g(α, y; ψ) ≡
g(α, y ∗ ; ψ), with y ∗ ′ = (y1∗ ′ , . . . , yn∗ ′ ).
The linear model representation allows the use of
computationally efficient methods for signal extraction and simulation smoothing. In particular, for the Monte Carlo evaluation of the likelihood as in (4), we require simulations from
g(θ |y; ψ) ≡ g(θ |y ∗ ; ψ) for a given set {b, C} where b and C
are functions of y and ψ. We have two options that lead to
numerically equivalent results. The first option is to apply the
state-space methods to model (9) as described by Durbin and
Koopman (2012). Signal extraction relies on the Kalman filter
smoother and simulation smoothing may rely on the methods of
de Jong and Shephard (1995) or Durbin and Koopman (2002).
The second option is to use a modification of the importance
sampling method of Jung, Liesenfeld, and Richard (2011) which
we refer to as the backward-forward (BF) method. We show in
Appendix A that the approach of Jung, Liesenfeld, and Richard
(2011) can be reshaped and extended to obtain a new algorithm
for signal extraction and simulation smoothing applied to model
(9). This completes the discussion of likelihood evaluation via
importance sampling. Several devices for simulation variance
reduction, including antithetic variables, can be incorporated
in the computations of (4). Next we discuss a new method for
choosing the importance parameter set {b, C} in (8). Existing
methods are reviewed in Section 2.5.
2.4 Numerically Accelerated Importance Sampling
The numerically accelerated importance sampling (NAIS)
method constructs optimal values for the importance parameter
set {b, C} using the same criterion for the efficient importance
sampling (EIS) method of Liesenfeld and Richard (2003) and
Richard and Zhang (2007). The values for b and C in the EIS
Koopman, Lucas, and Scharth: Numerically Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models
are chosen such that the variance of the log importance weights
log ω(θ, y; ψ) is minimized, that is,
min λ2 (θ, y; ψ)ω(θ, y; ψ)g(θ |y; ψ) dθ,
(11)
b,C
where
λ(θ, y; ψ) = log ω(y, θ ; ψ) = log p(y|θ ; ψ) − log g(y|θ ; ψ),
(12)
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
n
with g(y|θ ; ψ) = t=1 g(yt |θt ; ψ) and with g(yt |θt ; ψ) given
by (7). The minimization in (11) is high-dimensional and numerically not feasible in most cases of interest. The minimization in (11) is therefore reduced to a series of minimizations for
bt and Ct , for t = 1, . . . , n. Hence, we obtain the importance
parameters {bt , Ct } by
min λ2 (θt , yt ; ψ)ω(θt , yt ; ψ)g(θt |y; ψ) dθt , t = 1, . . . , n,
bt ,Ct
(13)
where ω(θt , yt ; ψ) = p(yt |θt ; ψ)/g(yt |θt ; ψ) and λ(θt , yt ; ψ) =
log ω(yt , θt ; ψ).
The NAIS method is based on the insight that the smoothing
density g(θt |y; ψ) ≡ g(θt |y ∗ ; ψ) is available analytically for the
linear Gaussian state-space model (9). For a given set {b, C} and
for a scalar signal θt , we have
1 −1
2
∗
g(θt |y ; ψ) = N(θt , Vt ) = exp − Vt (θt − θt ) / 2π Vt ,
2
117
obtain mean
θt and variance Vt from (14), for t = 1, . . . , n. New
values for bt and Ct are then obtained from the minimization
(15) that is reduced to a weighted least squares computation
θtj ; ψ) as the “dependent” varifor each t. We take log p(yt |
able, vector (1,
θtj , −0.5
θtj2 )′ as the “explanatory” variable and
θtj , yt ; ψ) as the “weight” for which all terms
exp(zj2 /2)h(zj )ω(
that do not depend on zj are dropped. The regression computations are based on sums over j = 1, . . . , M. The second
and third regression estimates, those associated with
θtj and
θtj2 ,
represent the new values for bt and Ct , respectively. The computations are repeated for each t or are done in parallel. Once
a new set for {b, C} is constructed, we can repeat this procedure. Convergence toward a final set {b, C} typically takes a few
iterations.
We start the procedure by having initial values for {b, C}. A
simple choice is bt = 0 and Ct = 1, for t = 1, . . . , n. We can
also initialize {b, C} by the local approximation of Durbin and
Koopman (1997); see Section 2.5. Finally, Richard and Zhang
(2007) argued that we can omit the term ω(
θtj , yt ; ψ) for computing the regression “weight” without much loss of numerical
efficiency. We prefer to adopt this modification because it is
computationally convenient. The NAIS procedure for selecting
the importance parameter set is reviewed in Figure 1.
2.5
Relation to Previous Methods
We show in Section 4 that the NAIS method is an efficient
method from both computational and numerical perspectives.
t = 1, . . . , n,
(14) We compare the NAIS with two related methods. The first is
the local approximation method of Shephard and Pitt (1997)
where the conditional mean
θt and variance Vt are obtained from and Durbin and Koopman (1997) which we refer to as SPDK. It
the Kalman filter and smoother (KFS) or the backward-forward is based on a second-order Taylor expansion of log p(yt |θt ; ψ)
(BF) smoothing method of Appendix A, in both cases applied around the conditional mode of p(θt |y; ψ) which can be computed iteratively using KFS or BF applied to model (9) for
to model (9).
θt and Vt as in the algorithm in Figure 1. The iterThis result allows us to directly minimize the low dimensional evaluating
integral (13) for each time period t via the method of numerical ations require analytic expressions for the first two derivatives
integration. For most cases of practical interest, the method of log p(yt |θt ; ψ), with respect to θt , for t = 1, . . . , n. The conproduces a virtually exact solution. Numerical integration is vergence toward the mode of p(θt |y; ψ) typically takes a few
reviewed in Monahan (2001). We adopt the Gauss–Hermite iterations; see Durbin and Koopman (2012).
The second method is the efficient importance sampling (EIS)
quadrature method that is based on a set of M abscissae zj
with associated weights h(zj ), for j = 1, . . . , M. The value of algorithm as developed by Richard and Zhang (2007). This
M is typically between 20 and 30. The values for the weights method approximates the minimization problem in (13) via the
h(zj ) are predetermined for any value of M. The Gauss–Hermite sampling of the signal θt = Zt αt via simulation smoothing from
quadrature approximation of the minimization (13) is then given g(α|y; ψ), for t = 1, . . . , n, rather than by numerical integration as in NAIS. In the modeling framework (1), the sampling is
by
carried out by the backward-forward sampler of Jung, LiesenM
feld, and Richard (2011). The BF sampler in Appendix A is a
2
min
h(zj ) exp(zj )ϕ(
θtj ),
(15)
modification and a more efficient implementation of this sambt ,Ct
j =1
pler. Furthermore, the introduction of antithetic variables halves
where ϕ(
θtj ) = λ2 (
θtj , yt ; ψ)ω(
θtj , yt ; ψ) g(
θtj |y ∗ ; ψ), with the simulation effort. Koopman, Lit, and Nguyen (2014) de1/2
θtj =
θt + Vt zj , for j = 1, . . . , M. The conditional mean
θt velop an implementation of EIS that is based on the KFS. We
and variance Vt are defined in Equation√(14) from which it fol- emphasize that the EIS uses simulation methods both for selectlows that g(
θtj |y ∗ ; ψ) = exp(−0.5zj2 )/ 2π Vt . Further details ing {b, C} and for evaluating the likelihood function. In Section
of our numerical implementation are provided in the online ap- 4 we consider the methods of SPDK and EIS using BF and
KFS. These methods for selecting the importance parameter set
pendix (supplementary materials).
The minimization (15) takes place iteratively as in the original are reviewed in detail in the online appendix (supplementary
EIS method. For a given importance parameter set {b, C}, we materials).
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
118
Journal of Business & Economic Statistics, January 2015
Figure 1. NAIS algorithm for selecting the importance parameter set {b, C}.
3.
NEW CONTROL VARIABLES FOR NAIS
We introduce a new set of control variables to improve the
numerical efficiency of likelihood evaluation using importance
sampling based on NAIS. We develop control variables that are
based on specific Taylor series expansions. The control variables can be evaluated by numerical integration using Gauss–
Hermite quadrature. We exploit the differences between the
estimates that are obtained from simulation and numerical integration methods with the sole purpose of reducing the variance
of importance sampling estimates. This approach of variance
reduction in the context of NAIS can replace the use of the
antithetic variables proposed by Ripley (1987) and Durbin and
Koopman (2000).
The
likelihood estimate (4) is the sample average ω¯ =
S −1 Ss=1 ωs multiplied by g(y; ψ), where
ωs = ω(θ (s) , y; ψ) =
t = 1, . . . , n,
n
ωts ,
t=1
ωts = ω(θt(s) , yt ; ψ),
s = 1, . . . , S,
with the importance sample weights ω(θt , yt ; ψ) defined below
(13), θ (s) generated from the importance density g(θ |y; ψ), and
θt(s) denoting the tth element of θ (s) , for t = 1, . . . , n. The density g(y; ψ) can be evaluated by the Kalman filter applied to
the linear Gaussian model (9) for some value of χ . The variance of the sample average ω¯ determines the efficiency of the
importance sampling likelihood estimate (4).
To reduce the variance of ω,
¯ we construct control variates
based on
x(θ, y; ψ) = log ω(θ, y; ψ) = log p(y|θ ; ψ) − log g(y|θ ; ψ).
The tth contribution of x(θ, y; ψ) is
given by x(θt , yt ; ψ) =
log ω(θt , yt ; ψ) such that x(θ, y; ψ) = nt=1 x(θt , yt ; ψ). Given
the draws θ (1) , . . . , θ (S) , we define
xs = log(ωs ) =
n
t=1
xts ,
s = 1, . . . , S,
where xts = log(ωts ), and hence ωts = exp(xts ) for t =
1, . . . , n. We can express the sample average of ωs in terms
of xs = log ωs by means of a Taylor series around some value
x, that is,
S
1
1
2
ω¯ = exp(x)
1 + [xs − x] + [xs − x] + · · · .(16)
S s=1
2
We adopt the terms involving xts , t = 1, . . . , n, in this expansion as control variables. Our method consists of replacing the
highest variance terms of the Taylor series by their probability
limits, which we compute efficiently via the NAIS algorithm.
This step clearly leads to a further reduction of the importance
sampling variance and to an improvement of the numerical efficiency at a low computational cost.
3.1 Construction of the First New Control Variable
We base our first control variable on the first order term
(xs − x) of the Taylor series expansion (16). Under the same
regularity conditions required for importance sampling, we have
x¯ =
S
1
p
xs −→
x,
S s=1
(17)
where
x = Eg x(θ, y; ψ) and where Eg is expectation with respect to density g(θ |y; ψ). The Taylor series expansion (16)
Koopman, Lucas, and Scharth: Numerically Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models
around x =
x can now be used to construct a first-order control
variable.
Since
x=
n
Eg [x(θt , yt ; ψ)] ,
t=1
we can evaluate
x by means of the Gauss–Hermite quadrature
method for each time index t separately as discussed in Section
2.4, that is,
xt = Eg [x(θt , yt ; ψ)] = x(θt , yt ; ψ)g(θt |y; ψ) dθt
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
≈
M
j =1
zj2
x(
θtj , yt ; ψ)g(
θtj |y; ψ)h(zj )e ,
1/2
where
θtj =
θt + Vt zj and with the numerical evaluation as
in (15). The Kalman filter and smoother computes
θt and Vt for
xt .
t = 1, . . . , n. Furthermore, we have
x = nt=1
The likelihood estimate (4) corrected for the first control
variable is given by
1
ψ)c = g(y; ψ) exp(
ωs − exp(
x )xs
x )
x+
L(y;
S s
ψ) + g(y; ψ) exp(
¯
= L(y;
x ) (
x − x).
Monte Carlo and empirical evidence of the importance of
the two control variables in reducing the simulation variance
of the likelihood estimate obtained from the NAIS method is
provided in Section 4 for the stochastic volatility model. More
evidence for other models is presented in the online appendix
(supplementary materials).
4.
4.1
MONTE CARLO AND EMPIRICAL EVIDENCE
FOR SV MODEL
Design of Monte Carlo Study
We consider the stochastic volatility (SV) model to illustrate the performance of NAIS in comparison to the alternative
importance sampling methods discussed in Section 2, with or
without the control variables introduced in Section 3. The SV
model may easily be one of the most widely studied nonlinear
state-space models. The references to some key developments
in the SV model literature are Tauchen and Pitts (1983), Taylor
(1986), and Melino and Turnbull (1990). Ghysels, Harvey, and
Renault (1996) and Shephard (2005) provided detailed reviews
of SV models.
For a time series of log-returns yt , we consider the k-factor
univariate stochastic volatility model of Liesenfeld and Richard
(2003) and Durham (2006), that is given by
yt ∼ N(0, σt2 ),
p
ψ)c −→ L(y; ψ). When
It follows from (5) and (17) that L(y;
the importance model (9) provides an accurate approximation
to the likelihood, ωs is close to one and xs is close to zero, such
that ωs ≈ 1 + xs . Hence, ωs and exp(
x )xs are typically highly
and positively correlated. When the importance model is a less
accurate approximation, the positive correlation remains, but at
ψ)c is a more efficient
a more moderate level. Therefore, L(y;
ψ).
estimate of the likelihood compared to L(y;
3.2 Construction of the Second New Control Variable
We base our second control variable on the second-order
term (xs − x)2 of the Taylor series expansion (16). We aim to
correct for the sample variation of (xts −
xt )2 within the sample
(1)
(S)
xt is the tth
of draws θt , . . . , θt for each t individually, where
element of
x . Using the same arguments as in Section 3.1, we
p
σt2 , where
have σ¯ t2 −→
σ¯ t2 =
σt2
1
(xts −
xt )2 ,
S
2
= Eg (xts −
xt ) =
(xts −
xt )2 g(θt |y; ψ) dθt .
We compute the variance
σt2 using the Gauss–Hermite quadrature. Define
n
ψ)cc = L(y;
ψ)c + 1 g(y; ψ) exp(
x)
(
σt2 − σ¯ t2 ),
L(y;
2
t=1
p
ψ)cc −→
L(y; ψ). Since we
from which it follows that L(y;
xt )2 by its probability
can replace the sample variation of (xts −
ψ)cc to be more efficient than
limit, we can expect estimate L(y;
ψ) and L(y;
ψ)c .
L(y;
119
σt2 = exp(c + θt ),
θt = α1,t + · · · + αk,t ,
t = 1, . . . , n,
with scalar constant c, scalar signal θt and k × 1 state vector
αt = (α1,t , . . . , αk,t )′ . The state vector is modeled by (1) with
k × k diagonal time-invariant matrices Tt and Qt given by
⎤
⎡ 2
⎡
⎤
ση,1 0
0
0
φ1 0
⎥
⎥
⎢
⎢
..
..
⎥
Qt = ⎢
Tt = ⎢
. 0⎥
.
0 ⎦,
⎦,
⎣ 0
⎣0
0
0
φk
0
0
2
ση,k
2
and with unknown coefficients |φi | < 1 and ση,i
> 0, for i =
1, . . . , k. We identify the parameters by imposing φ1 > · · · >
φk . The signal θt represents the log-volatility.
For the purpose of likelihood evaluation by importance sampling, we examine the performance of the SPDK method, two
implementations of the standard EIS method and four implementations of our NAIS method. The SPDK method is based on
the mode approximation that is computed iteratively using the
Kalman filter and smoother (KFS); see Section 2.5. The likelihood function is evaluated using the simulation smoother of
de Jong and Shephard (1995) or Durbin and Koopman (2002)
which we refer to as JSDK. The EIS method is based on a Monte
Carlo approximation to obtain the importance parameter set via
the minimization (13); see Section 2.5. Our proposed NAIS
method is introduced in Section 2.4. The likelihood evaluation
using EIS and NAIS can both be based on the JSDK simulation
smoother or the BF sampler of Appendix A. Finally, both NAIS
implementations can be extended with the use of the new control
variables introduced in Section 3. Table 1 reviews the methods
and their different implementations.
The design of the Monte Carlo study is as follows. We consider 500 random time series for each of the two SV models in
120
Journal of Business & Economic Statistics, January 2015
Table 1. Importance sampling methods for state-space models
I
Task
SPDK
EIS-BF
EIS-JSDK
IS parameter set {b, C} via
constructed by simulation
or by numerical integration
using method
Local mode
—
—
KFS
(13)
√
(13)
√
—
BF
—
JSDK
JSDK
BF
JSDK
—
—
—
II
Simulation method
III
Control variables (optional)
NAIS-BF
(-Ctrl)
NAIS-JSDK
(-Ctrl)
(13)
—
√
(13)
—
√
BF
KFS
BF
√
( )
JSDK
√
( )
NOTE: Likelihood evalution via importance sampling: I, obtain importance sampling (IS) parameter set {b, c} using methods described in Section 2.4 and 2.5; II, Sampling from the
importance density to construct likelihood function; III, in case of NIAS, with use of control variable in Section 3 or not.
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
our study. The first SV model has a single log-volatility component, k = 1, with coefficients
c = 1,
φ1 = 0.98,
2
ση,1
= 0.0225,
which are typical values in empirical studies for daily stock
returns. The second SV model has k = 2 with
c = 1,
φ2 = 0.9,
φ1 = 0.99,
2
ση,1
= 0.005,
2
ση,2
= 0.03.
We take 500 simulated time series to avoid the dependence of
our conclusions on particular trajectories of the simulated states
and series. For each simulated time series, we estimate the loglikelihood function at the true parameters a hundred times using
different common random numbers. Hence, each cell in the
tables presented below reflects 50,000 likelihood evaluations.
We report the results for two different sample sizes, n = 1000
and n = 5000, and we use S = 200 importance samples for each
likelihood evaluation and for each method. The number of nodes
for the numerical integration calculations is set to M = 20.
We start by estimating the variance and the bias associated
with each importance sampling method. We compute the reported statistics as
Bias =
Variance =
500
100
1
j (y i ; ψ) − log L(y i ; ψ) ,
·
log L
50, 000 i=1 j =1
500
100
1 1
j (y i ; ψ)
·
·
log L
500 i=1 100 j =1
2
− log L(y i ; ψ) ,
(18)
where y i is the ith simulated time series, log L(y i ; ψ) is the
j (y i ; ψ) is the estimate of
“true” log-likelihood value, log L
the log-likelihood function for a particular method and for
the jth set of common random
numbers, j = 1, . . . , 100,
log
Lj (y i ; ψ). The true logand log L(y i ; ψ) = 100−1 100
j =1
likelihood value is unknown but we take its “true” value as
the log of the average of likelihood estimates from the NAIS
method with S = 200 × 100 = 20,000 importance samples.
We expect the approximation error with respect to the true
likelihood to be small. We compute the mean squared error
(MSE) as the sum of the variance and the squared bias. The
variance and the MSE are reported as a ratios with respect to
the EIS-BF method; see Table 1.
The numerical efficiency of estimation via simulation can be
increased by generating additional samples. A systematic comparison between methods must therefore take into account computational efficiency. We report the average computing times
for each method on the basis of an Intel Duo Core 2.5GHz
processor. We record the times required for constructing the importance sampling parameter set (task I) and for jointly generating importance samples and computing the likelihood estimate
(task II). In case of NAIS with control variables, we include
the additional time for task III in the total time of task II. Our
key summary statistic is the time normalized variance ratio of
method a against the benchmark method b and it is given by
−1
TimeaI+II − TimebI+II
,
(19)
Variancea/b × 1 +
TimeaII
where Variancea/b is the ratio of the variance defined in (18)
for methods a and b and Timem
j is the time length of task j, for
j = I, II, I+II, by method m, for m = a, b. We have excluded
TimeaI from the denominator because it is a fixed cost and not
relevant for drawing additional samples.
To have the computing times of EIS and NAIS implementations comparable to each other, we initialize the minimization of
(11) by the local approximation for {b, C} of the SPDK method.
To reduce the simulation variance for all methods, we use antithetic variables for location as in Durbin and Koopman (2000)
except for the NAIS methods that include the control variables
of Section 3. We have found no evidence of importance sampling weights that constitute an infinite variance in our study;
see the discussions in Koopman, Shephard, and Creal (2009).
Our diagnostic procedure includes the verification of how sensitive the importance sampling weights are to artificial outliers as
in Richard and Zhang (2007). We have efficiently implemented
all methods using MATLAB and C.
4.2 Monte Carlo Results: Log-Likelihood Estimation
Table 2 presents the results for the stochastic volatility model
with k = 1. The results show that our numerical integration
method for constructing the importance density leads to 40%–
50% reductions in the variance of the log-likelihood estimates
compared to the EIS method. In all cases the numerical gains
become larger when we increase the time series dimension from
n = 1000 to n = 5000. The use of the control variable further
increases the efficiency of the NAIS method as it generates a
further 20% –35% gain in producing a smaller variance. We find
no deterioration in relative performance for the MSE measure,
Koopman, Lucas, and Scharth: Numerically Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models
121
Table 2. Log-likelihood errors for stochastic volatility model, k = 1
n = 1000, S = 200
SPDK
EIS-BF
EIS-JSDK
NAIS-BF
NAIS-JSDK
NAIS-BF-Ctrl
NAIS-JSDK-Ctrl
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
n = 5000, S = 200
SPDK
EIS-BF
EIS-JSDK
NAIS-BF
NAIS-JSDK
NAIS-BF-Ctrl
NAIS-JSDK-Ctrl
Variance
MSE
Time step 1
(×10)
Time step 2
(×10)
TNVAR
12.779
1.000
1.009
0.595
0.594
0.405
0.415
12.837
1.000
1.009
0.595
0.594
0.406
0.416
0.022
0.232
0.247
0.073
0.070
0.073
0.073
0.173
0.187
0.172
0.174
0.175
0.192
0.180
5.576
1.000
1.005
0.299
0.297
0.224
0.216
Variance
MSE
Time step 1
(×10)
Time step 2
(×10)
TNVAR
15.025
1.000
0.997
0.503
0.501
0.380
0.375
18.594
1.000
0.998
0.501
0.499
0.381
0.376
0.052
1.278
1.246
0.364
0.340
0.418
0.398
0.908
0.990
0.908
0.928
0.908
1.002
0.939
6.152
1.000
0.885
0.245
0.236
0.206
0.189
NOTE: The table presents the numerical and computational performance of different IS methods for log-likelihood estimation. We simulate 500 different realizations from the model.
For each of these realizations, we obtain log-likelihood estimates for 100 different sets of random numbers. We estimate the variance associated with each method as the average sample
variance across the 500 realizations. We define the mean-square error (MSE) as the sum of the variance and the square of the average bias across the 500 realizations. We show these
statistics as ratios with the standard implementation of the EIS-BF method as the benchmark. The time for step 1 column gives the fixed time cost for obtaining the parameters of the
importance density, while the time for step 2 refers to the computational cost of sampling from the importance density and calculating the likelihood estimate. The TNVAR column reports
the time normalized variance ratio according to (19). Table 1 lists the methods used and their acronyms. NAIS-BF-Ctrl and NAIS-JSDK-Ctrl refer to the NAIS methods with use of the
control variables of Section 3. We specify the stochastic volatility model as: yt ∼ N(0, σt2 ) with σt2 = exp(αt ) and αt+1 = 0.98αt + ηt where ηt ∼ N(0, ση2 = 0.0225) for t = 1, . . . , n.
confirming the accuracy of the numerical integration method for
obtaining the control variables.
The results further show that the NAIS method can also
achieve substantial gains in computational efficiency. It is able
to construct the importance density 70% faster than the EIS
method. This result is partly due to the ability of working with
the marginal densities and the scope for optimizing the computer
code in this setting. In all cases the EIS method takes longer to
construct the importance density than it does to generate the
importance samples and to compute the log-likelihood estimate
based on S = 200 simulation samples. When we normalize the
variances by the length of computing time, we also obtain gains
for the NAIS method. Table 2 presents a total improvement of
70%–80% by the NAIS method with control variables. The gain
in performance is comparable when the default EIS method is
compared with the SPDK method. The results further indicate
that the computing times for the BF and JSDK simulation methods are equivalent for the SV model with k = 1.
Table 3 reports the findings for the SV model with two logvolatility components, k = 2. These results are overall similar
Table 3. Log-likelihood errors for stochastic volatility model, k = 2
n = 1000, S = 200
SPDK
EIS-BF
EIS-JSDK
NAIS-BF
NAIS-JSDK
NAIS-BF-Ctrl
NAIS-JSDK-Ctrl
n = 5000, S = 200
SPDK
EIS-BF
EIS-JSDK
NAIS-BF
NAIS-JSDK
NAIS-BF-Ctrl
NAIS-JSDK-Ctrl
Variance
MSE
Time step 1
(×10)
Time step 2
(×10)
TNVAR
17.523
1.000
1.011
0.562
0.562
0.374
0.374
17.539
1.000
1.001
0.564
0.554
0.378
0.373
0.223
0.763
0.578
0.569
0.364
0.592
0.385
0.238
0.374
0.314
0.304
0.237
0.408
0.307
4.572
1.000
0.569
0.301
0.172
0.280
0.153
Variance
MSE
Time step 1
(×10)
Time step 2
(×10)
TNVAR
18.915
1.000
0.972
0.483
0.474
0.368
0.365
26.283
1.000
0.963
0.484
0.465
0.371
0.363
1.341
3.988
3.067
3.010
1.976
3.089
2.050
1.293
1.962
1.673
1.619
1.298
2.152
1.655
5.305
1.000
0.564
0.266
0.155
0.277
0.155
NOTE: We refer to the description of Table 2. We specify the two-factor stochastic volatility model as: yt ∼ N(0, σt2 ) with σt2 = exp(αt,1 + αt,2 ), αt+1,1 = 0.99αt,1 + ηt,1 , αt+1,1 =
2
2
0.9αt,2 + ηt,2 , where ηt,1 ∼ N(0, ση,1
= 0.005), ηt,2 ∼ N(0, ση,2
= 0.03), for t = 1, . . . , n.
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
122
Journal of Business & Economic Statistics, January 2015
to the ones for the SV model with k = 1 in Table 2. However, we
find two important differences. First, for a model with multiple
states, as in the SV model with k = 2, we achieve a small gain
in computational efficiency by switching from the BF sampler
to the JSDK sampler. The main reason is that JSDK is able to
simulate the univariate signal θt directly whereas the BF sampler
needs to simulate the signal via the (multiple) state vector αt .
Hence, we obtain around 83% reductions in the variance normalized by time when using the NAIS-JSDK method instead of
the EIS-BF method.
The second difference between the results for the SV model
with k = 1 and k = 2 concerns the variance reduction that we
obtain by using control variables in the likelihood estimation.
For the case of k = 2, the variance reductions due to the
control variables are similar to those for k = 1, but the relative
increase in the computational cost for step 2 is higher. This is
because antithetic variables reduce the computational effort for
simulation.
4.3 Parameter Estimation for the SV Model Using NAIS
To illustrate the performance of our proposed NAIS method
in more detail, we consider the simulated maximum likelihood
estimation of the unknown parameters in a multifactor stochastic
volatility model. We report our findings from both a Monte Carlo
study and an empirical study below. We carry out parameter
estimation using the following consecutive steps:
1. Set a starting value for the estimate of parameter vector ψ.
2. Set starting values for the importance sampling parameter χ
in (8).
3. Maximize the log-likelihood function with respect to ψ using
the NAIS-JSDK method with S = 0 and only the first control
variate; use the estimates as starting values for the next step.
4. Maximize the log-likelihood function with respect to ψ using
the NAIS-JSDK-Ctrl method with S > 0.
The estimation in Step 3 concerns an approximate loglikelihood function. It is fast and requires no simulation, only
numerical integration. The computational efficiency of this procedure is primarily due to the accurate approximation of the
log-likelihood function calculated by the NAIS-JSDK method
with S = 0 and the first control variate from Section 3. As a result, the convergence of the maximization in the last step is fast
as it only requires a small number of iterations. The consequence
is that we can set S at a high value in Step 4 as it only marginally
increases the required computing time. The maximizations in
Steps 3 and 4 are based on a quasi-Newton method. The use
of common random numbers in Step 4 for each likelihood
estimation leads to a smooth likelihood function in ψ, which
is necessary for the application of the quasi-Newton optimization methods. In both studies below we set S = 200 in
Step 4.
4.3.1 Monte Carlo Evidence. We consider the k-factor
stochastic volatility model with k = 3 for time series of lengths
n = 5000 and n = 10,000. The true parameter values are
2
= 0.005,
set to d = 0.5, φ1 = 0.99, φ2 = 0.9, φ3 = 0.4, ση,1
2
2
ση,2 = 0.016, and ση,3 = 0.05. We take these parameter values
also as the starting values for parameter estimation. We draw 100
Table 4. Parameter estimation for stochastic volatility model, k = 3
n = 5000
c
φ1
2
σ1,η
φ2
2
σ2,η
φ3
2
σ3,η
True
Mean
MC
Std. error
Total
Std. error
MC variance
Ratio
0.500
0.990
0.005
0.900
0.015
0.400
0.050
0.489
0.989
0.005
0.883
0.014
0.314
0.054
0.003
0.000
0.000
0.009
0.001
0.025
0.003
0.098
0.005
0.002
0.055
0.007
0.224
0.032
0.001
0.010
0.007
0.029
0.009
0.012
0.006
n = 10,000
c
φ1
2
σ1,η
φ2
2
σ2,η
φ3
2
σ3,η
True
Mean
MC
Std. Error
Total
Std. Error
MC variance
Ratio
0.500
0.990
0.005
0.900
0.015
0.400
0.050
0.500
0.990
0.005
0.893
0.015
0.334
0.054
0.003
0.000
0.000
0.005
0.001
0.061
0.004
0.078
0.003
0.002
0.034
0.005
0.205
0.023
0.001
0.008
0.010
0.021
0.046
0.090
0.035
NOTE: We simulate 100 trajectories of a three-factor stochastic volatility model. For each of these realizations, we obtain 20 simulated maximum likelihood parameter estimates based
on different sets of common random numbers and using the NAIS-JSDK method. We first show the average parameter estimates across the 500 replications. The Monte Carlo (MC)
standard error column reports the square root of the average of the sample variance of the parameter estimates across the 100 realizations. The total standard error column shows the
square-root of the sum of the MC variance and the variance of the average estimates across the 100 trajectories. Finally, we obtain the MC variance ratio by dividing the MC variance by
the total variance. The starting values are the true parameters. The average computing time was 65s and 123s for n = 5000 and n = 10,000, respectively. yt ∼ N(0, σt2 ), t = 1, . . . , n,
σt2 = exp(θt ), θt = d + α1,t + α2,t + α3,t , αt = T αt−1 + ηt , α1 ∼ N(a1 , P1 ), ηt ∼ N(0, Q), where T is a diagonal matrix with elements φ1 , φ2 , and φ3 and Q is a diagonal matrix with
2
2
2
elements ση,1
, ση,2
, and ση,3
.
Koopman, Lucas, and Scharth: Numerically Accelerated Importance Sampling for Nonlinear Non-Gaussian State-Space Models
123
Table 5. Two-factor stochastic volatility model: Empirical study
GE
Downloaded by [Universitas Maritim Raja Ali Haji] at 19:19 11 January 2016
NAIS
JP Morgan
EIS
NAIS
EIS
Coca-Cola
NAIS
EIS
AT&T
NAIS
EIS
Wal-Mart
NAIS
EIS
Exxon
NAIS
EIS
c
0.845
0.845
[0.002] [0.006]
(0.690)
1.275
1.274
[0.003] [0.005]
(0.458)
0.004
0.005
[0.004] [0.007]
(0.393)
0.655
0.655
[0.001] [0.002]
(0.652)
0.375
0.374
[0.002] [0.003]
(0.413)
0.555
0.555
[0.000] [0.001]
(0.193)
φ1
0.997
0.997
[0.000] [0.000]
(0.002)
0.997
0.997
[0.000] [0.000]
(0.002)
0.995
0.995
[0.000] [0.001]
(0.003)
0.997
0.997
[0.000] [0.000]
(0.002)
0.996
0.996
[0.000] [0.000]
(0.002)
0.991
0.991
[0.000] [0.000]
(0.006)
2
σ1,η
0.007
0.007
[0.000] [0.000]
(0.003)
0.008
0.008
[0.000] [0.000]
(0.003)
0.006
0.006
[0.000] [0.001]
(0.003)
0.005
0.005
[0.000] [0.000]
(0.003)
0.005
0.005
[0.000] [0.000]
(0.002)
0.008
0.008
[0.000] [0.001]
(0.008)
φ2
0.458
0.455
[0.016] [0.042]
(0.271)
0.839
0.840
[0.004] [0.006]
(0.065)
0.488
0.484
[0.037] [0.076]
(0.164)
0.897
0.898
[0.002] [0.004]
(0.081)
0.275
0.274
[0.013] [0.028]
(0.118)
0.936
0.936
[0.000] [0.002]
(0.048)
2
σ2,η
0.261
0.261
[0.008] [0.017]
(0.091)
0.086
0.085
[0.002] [0.003]
(0.032)
0.307
0.305
[0.017] [0.029]
(0.077)
0.027
0.027
[0.000] [0.001]
(0.018)
0.263
0.262
[0.007] [0.009]
(0.054)
0.026
0.026
[0.000] [0.001]
(0.013)
86
–65%
67
–66%
69
–52%
74
–65%
36
–51%
46
–47%
Time (seconds)
Improvement
244
196
146
212
73
86
NOTE: We estimate a two-component stochastic volatility specification for the daily returns of six Dow Jones index stocks in the period between January 2001 and December 2010 (2512
observations). We repeat the NAIS-JSDK and EIS-BF estimation methods a hundred times with different random numbers. The Monte Carlo and statistical standard errors are in brackets
[ ] and parenthese