(c) Show that if 7rl , ..• , 7r m are nonnegative, satisfy the balance equations, and sum

to 1 , then

{ 0, if i is transient,

if i is recurrent, 7ri =

400 Markov Chains Chap. 7

where t; is the mean recurrence time of i.

(d) Show that the distribution of part (b) is the unique probability distribution that

satisfies the balance equations. Note: This problem not only provides an alternative proof of the existence and unique

ness of probability distributions that satisfy the balance equations, but also makes an intuitive connection between steady-state probabilities and mean recurrence times. The main idea is to break the process into "cycles," with a new cycle starting each time that the recurrent state is visited. The steady-state probability of can be interpreted as

the long-term expected frequency of visits to state s which is inversely proportional to s the average time between consecutive visits (the mean recurrence time); cf. part (c). s, Furthermore, if a state i is expected to be visited, say, twice as often as some other state j during a typical cycle, it is plausible that the long-term expected frequency 7ri of state i will be twice as large as 7rj . Thus, the steady-state probabilities 7ri should be proportional to the expected number of visits Pi during a cycle; cf. part (b).

Xn,

Xo s.

Pi = 00 L # # = i).

We claim that for all

Solution. (a) Consider the Markov chain

initialized with

P(X1 s, ... , Xn-1 s, Xn n=l

number of visits to state i before the next visit to state s is equal to 0

To see this, we first consider the case i # and let

be the random variable that takes the value 1 if

Xl Xn-1 s, Xn

# S, ••• ,

and

i, and the value otherwise. Then, the

1 In.

[00 ] 00 DC = = # S, . . .

Thus,t

E �In �E[Inl �P(X1 ,Xn-1 s,Xn

= i).

t The interchange of the infinite summation and the expectation in the subsequent calculation can be justified by the following argument. We have for any k > 0,

Let

be the first positive time that state is visited. Then,

= t) is equal to zero. We take the limit of both sides of the earlier 2: 1 P(T equation, as 2:�k+2 P(T k -+ 00, to obtain the desired relation

Since the mean recurrence time

= t) is finite, the limit, as k -+ 00 of

Problems 401 When i = s, the events

{Xl '# s, . . . ,Xn - l '# s, Xn = s},

for the different values of n, form a partition of the sample space, because they corre spond to the different possibilities for the time of the next visit to state s. Thus,

L :x:

P(XI '# S, ••• , Xn - l '# s, Xn = s) = 1 = ps , n= l

which completes the verification o f our assertion. We next use the total probability theorem to write for n � 2,

P(XI '# S, ••• , Xn- l '# s, Xn i) =

P (XI '# S, . .. , Xn-2 '# s, Xn- 1 L = k)Pki . ki's

We thus obtain

P(XI '# S, L ••. , Xn-l '# s, Xn = i) n=l

Pi =

L P(XI '# S, , Xn- 1 '# s, Xn = i) n=2

= Psi +

P(XI '# S, ••• , Xn -2 '# S, Xn- 1 LL = k)Pki n=2 ki's

:x:

= Psi +

= Psi + L Pki L P(X1 ,# s, . . . ,Xn-2 ,# s, Xn- 1 = k)

ki's n=2

(c) Show that if 7rl , ..• , 7r m are nonnegative, satisfy the balance equations, and sum