Stochastic Processes and their Applications 92 2001 131–162 www.elsevier.comlocatespa
Large deviations for the Fleming–Viot process with neutral mutation and selection, II
Donald A. Dawson
a; ∗
, Shui Feng
b
a
The Fields Institute, 222 College Street, Toronto, ONT, Canada M5T 3J1
b
Department of Mathematics and Statistics, McMaster University, Hamilton, ONT, Canada L8S 4K1 Received 5 January 2000; received in revised form 10 July 2000; accepted 6 August 2000
Abstract
Large deviation principles are established for the Fleming–Viot process with neutral mutation and with selection, and the associated equilibrium measures as the sampling rate approaches zero
and when the state space is equipped with the weak topology. The path-level large deviation results improve the results of Dawson and Feng 1998, Stochastic Process. Appl. 77, 207–232 in
three aspects: the state space is more natural, the initial condition is relaxed, and a large deviation principle is established for the Fleming–Viot process with selection. These improvements are
achieved through a detailed study of the behaviour near the boundary of the Fleming–Viot process with nite types. c
2001 Elsevier Science B.V. All rights reserved. MSC:
primary 60F10; secondary 92D10 Keywords:
Fleming–Viot process; Large deviations; Weak and -topology
1. Introduction
The Fleming–Viot process henceforth, FV process is a probability-valued stochas- tic process describing the evolution of the distribution of genotypes in a population
under the in
uence of mutation, replacement sampling, and selective advantages among various genotypes.
Let E = [0; 1], CE be the set of continuous functions on E, and M
1
E denote the space of all probability measures on E equipped with the usual weak topology
and Prohorov metric . Let A be the generator of a Markov process on E with do- main DA. Dene D =
{F: F=fh; i; f ∈ C
∞ b
R; ∈ DA; ∈ M
1
E }, where
C
∞ b
R denotes the set of all bounded, innitely dierentiable functions on R. Then the generator of the FV process in this article has the form
L F =
Z
E
A F
x d x +
2 Z
E
Z
E 2
F xy
Q; d x; dy
Research supported by the Natural Science and Engineering Research Council of Canada.
∗
Corresponding author. E-mail
addresses: donelds.elds.utoronto.ca
D.A. Dawson,
shuifengmcmail.cis.mcmaster.ca S. Feng.
0304-414901 - see front matter c
2001 Elsevier Science B.V. All rights reserved. PII: S 0 3 0 4 - 4 1 4 9 0 0 0 0 0 7 0 - 3
=f
′
h; ih; Ai + 2
Z Z f
′′
h; ixyQ; dx; dy; 1
where F=x = lim
→0+ −1
{F1 − +
x
− F};
2
F=xy= lim
1
→0+;
2
→0+ 1
2 −1
{F1 −
1
−
2
+
1 x
+
2 y
−F}; Q; d x; dy = dx
x
dy − dxdy;
and
x
stands for the Dirac measure at x ∈ E. The domain of L is D. E is called the
type space or the space of alleles, A is known as the mutation operator, and the last term describes the continuous sampling with sampling rate
. If the mutation operator
has the form
Afx = 2
Z fy
− fx dy
with ∈ M
1
E, we call the process a FV process with neutral mutation. For any symmetric bounded measurable function V z; y on E
⊗2
, let V =
Z
E
Z
E
V z; y d z dy and
V ;
F =
Z
E
Z
E
Z
E
F z[V z; y
− V y; w] dzdydw: Then the generator of a FV process with neutral mutation and selection takes the
form L
V
F = L F +
V ;
F ;
1.2 where V is called the tness function which is assumed to be continuous in the sequel.
A nice survey on FV process and their properties can be found in Ethier and Kurtz 1993. In particular, it is shown in Ethier and Kurtz 1993 that the martingale problem
associated with generators L and L
V
are well-posed. Let T ¿ 0 be xed, and C[0; T ]; M
1
E denote the space of all M
1
E-valued, continuous functions on [0; T ]. For any in M
1
E, let P
;
;
and P
;
; V;
be the laws of the FV process with neutral mutation and FV process with neutral mutation and
selection, respectively.
;
;
and
;
; ;V
will represent the corresponding equilibrium measures.
Let XE be the space of all nitely additive, non-negative, mass one measures on E, equipped with the smallest topology such that for all Borel subset B of E,
B is continuous in . The -algebra B of space XE is the smallest -algebra such that for all Borel subset B of E, B is a measurable function of . It is clear
that M
1
E is a strict subset of XE. In Dawson and Feng 1998, large deviation principle henceforth, LDP is established for equilibrium measures on space XE and
partial results are obtained for the path-level LPDs on a strange space under stronger topologies. In the present article we will rst establish the LDPs for the equilibrium
measures on space M
1
E, and compare these with results obtained in Dawson and Feng 1998. Secondly, we establish the LDPs for the FV process with neutral mutation and
with selection as → 0 on space C[0; T ]; M
1
E. These improve the corresponding results in Dawson and Feng 1998 in three aspects: the space is more natural, the
initial condition is relaxed, and a full large deviation principle is established for the FV process with selection. This type of LDPs can be viewed as the innite-dimensional
generalization of the Freidlin–Wentzell theory. We prove the results through a detailed analysis of the boundary behaviour of the FV process with nite types.
In Section 2, we list some preliminary results to make the paper self-contained. LDPs for the equilibrium measures are the content of Section 3. The detailed study of
the Fleming–Viot process with nite type space is carried out in Section 4. Finally, in Section 5, we prove the LDPs for the FV process with neutral mutation and with
selection.
2. Preliminary
