(a) For any transient state i, let ai (k) be the probability that starting from i we

will reach a state in the kth recurrent class. Derive a system of equations whose solution are the ai (k).

(b) Show that each of the n-step transition probabilities nj (n) converges to a limit,

and discuss how these limits can be calculated. Solution. (a) We introduce a new Markov chain that has only transient and absorbing

states. The transient states correspond to the transient states of the original, while the absorbing states correspond to the recurrent classes of the original. The transition probabilities Pij of the new chain are as follows: if i and j are transient, Pij = Pij ; if i is

a transient state and k corresponds to a recurrent class, Pik is the sum of the transition probabilities from i to states in the recurrent class in the original Markov chain. The desired probabilities ai (k) are the absorption probabilities in the new Markov chain and are given by the corresponding formulas:

ai (k) = Pik + L Pijaj (k), for all transient i.

j: transient

(b) If i and j are recurrent but belong to different classes, nj (n) is always o. If i

and j are recurrent but belong to the same class, Tij (n) converges to the steady-state probability of j in a chain consisting of only this particular recurrent class. If j is transient, Tij (n) converges to O. Finally, if i is transient and j is recurrent, then Tij (n) converges to the product of two probabilities: (1) the probability that starting from i we will reach a state in the recurrent class of j, and (2) the steady-state probability of

j conditioned on the initial state being in the class of j. Problem 33. * Mean first passage times. Consider a Markov chain with a single

recurrent class, and let s be a fixed recurrent state. Show that the system of equations

ts 0,

ti = 1+

L Pijtj , for all i f: s,

j=l

satisfied by the mean first passage times, has a unique solution. Hint: If there are two solutions, find a system of equations that is satisfied by their difference, and look for

its solutions.

Problems 399 Solution. Let ti be the mean first passage times. These satisfy the given system of

equations. To show uniqueness, let Ii be another solution. Then we have for all i =I- s

ti = 1+ L Pijtj , Ii = 1+ L PijIj ,

j#s

and by subtraction, we obtain

8i = L Pij8j ,

j#s

where 8i = Ii - ti. By applying m successive times this relation, if follows that

8i = L PiiI L PiIh · · · L Pjm-Iim . 8jm ·

Hence, we have for all i =I- s

= P (X I =I- s, . . . , Xm =I- s l Xo = i) · max l8j l · j

On the other hand, we have P(XI =I- s, . . . , Xm =I- s I Xo =

i) < 1 . This is because

starting from any state there is positive probability that s is reached in m steps. It follows that all the 8i must be equal to zero.

Problem 34. * Balance equations and mean recurrence times. Consider a Markov chain with a single recurrent class, and let s be a fixed recurrent state. For any state i, let

Pt = E [N umber of visits to i between two successive visits to s ] ,

where by convention, ps = 1. (a) Show that for all i, we have

(a) For any transient state i, let ai (k) be the probability that starting from i we