(b) Show that if the states of the two chains became equal by time n, their occupancy

probabilities at time n are the same, that is,

P(Xn = j IT � n) = P(Yn = j IT � n). (c) Show that ITij (n) - Tkj (n) 1 � C"Yn , for all i, j, k, and n. Hint: Condition on the

two events {T > n} and {T � n}.

(d) Let q j ( n ) = maxi Tij (n) and qj ( n ) = mini Tij (n). Show that

for all n.

(e) Show that the sequence Tij (n) converges to a limit that does not depend on i.

Hint: Combine the results of parts (c) and (d) to show that the two sequences qj (n) and qj (n) converge and have the same limit.

Solution. (a) The argument is similar to the one used to bound the PMF of the time until a recurrent state is entered (Problem 7). Let

I be some recurrent state and let

{3 = mini Til (n) > O. No matter what is the current state of Xn and Yn , there is probability of at least {32 that both chains are at state l after n time steps. Thus,

Similarly, and

388 Markov Chains Chap. 7 This implies that

P( T � n) � ern ,

where 'Y = (1 and c = 1/(1 - (32)I/n, - (32 )'n.

( b ) We condition on the possible values of T and on the common state l of the two chains at time T, and use the total probability theorem. We have

P(Xn = j1T � n) = �� P(Xn = j1T = t, Xt = l) P( T = t, Xt = I IT � n)

t=O 1=1 n m

= �� P(Xn = j1 Xt = l) P( T = t, Xt = � n) II T

t=O 1=1 n m

= � � rlj (n - t) P( T = t, Xt =

� n). II T

t=o 1=1

Similarly,

P(Yn = j1 T � n) = � � rlj (n - t) P( T = t, Yt = I I T � n).

t=o 1=1

But the events {T = t, Xt = l} and {T = t, Yt = l} are identical, and therefore have the same probability, which implies that P(Xn = j1T � n) = P(Yn = j1 T � n).

( c ) We have

rij(n) = P(Xn = j) = P(Xn = j1 T � n) P( T � n) + P(Xn = j1 T > n) P( T > n)

and

rkj (n) = P(Yn = j) = P(Yn = j1 T � n) P( T � n) + P(Yn = j1 T > n) P( T > n).

By subtracting these two equations, using the result of part ( b ) to eliminate the first terms in their right-hand sides, and by taking the absolute value of both sides, we obtain

Irij (n) - rkj (n)1 � Ip(Xn = j1 T > n) P( T > n) - P(Yn = j1 T > n) P( T > n) 1 � P( T > n) � er n.

( d) By conditioning on the state after the first transition, and using the total probability theorem, we have the following variant of the Chapman-Kolmogorov equation:

rij (n + 1) = � Pikrkj (n). k =1

Problems 389 Using this equation, we obtain

qt(n + 1) = m�x TiJ (n + 1) = max PikTkj (n) ::::; m� Pikqt (n) = qt (n).

k=l

The inequality q; (n) ::::; q; (n + 1) is established by a symmetrical argument. The inequality q; (n + 1) ::::; qt (n + 1) is a consequence of the definitions.

( e) The sequences q; (n) and qt (n) converge because they are monotonic. The inequal ity ITij (n) - TkJ (n)1 ::::; e,n , for all i and k, implies that qj (n) - qj (n) ::::; ern. Taking

the limit as n ---+ 00, we obtain that the limits of qj (n) and qj (n) are the same. Let 1T'j denote this common limit. Since qj (n) ::::; Tij(n) ::::; qj(n), it follows that TiJ (n) also

converges to 1T'j , and the limit is independent of i. Problem 20.* Uniqueness of solutions to the balance equations. Consider a

Markov chain with a single recurrent class, plus possibly some transient states. ( a) Assuming that the recurrent class is aperiodic, show that the balance equations

together with the normalization equation have a unique nonnegative solution. Hint: Given a solution different from the steady-state probabilities, let it be the PMF of Xo and consider what happens as time goes to infinity.

(b) Show that if the states of the two chains became equal by time n, their occupancy