( b ) Find the recurrent and the transient states. (c ) Find the expected number of years for the professor to achieve the highest ranking

given that in the first year he achieved the ith ranking.

Consider the !vlarkov s pe c i fie d . The steady-state p robabi li t ie s are

71"1 == - .

== -.

u 1 j s t b e fo r e t h e fi r s t t r a n si t i o n . ( a )

Assume that the process is in s t a te

in state 1 sixth tr an s iti on?

t hat

process will

(b) Determine the expected value and variance of the number of transitions up to

n e xt

process ret urns to state 1.

Consi der t he 1\I arkov chai n specified in Fig. 7.25.

Transition

for Problem 30.

(a) transient and recurrent states. Also. determine the recurrent classes

ones. if

are

(b) Do there exist steady-state probabilities given t he s tarts

state I?

are (c)

exist steady-state probabilities the process starts state

are

Problems 397

(d) Assume that the process starts in state 1 but we begin observing it after it reaches

steady-state. (i) Find the probability that the state increases by one during the first transi

tion we observe. (ii) Find the conditional probability that the process was in state 2 when we

started observing it, given that the state increased by one during the first transition that we observed.

(iii) Find the probability that the state increased by one during the first change of state that we observed.

( e) Assume that the process starts in state 4. (i) For each recurrent class, determine the probability that we eventually reach

that class. (ii) What is the expected number of transitions up to and including the tran

sition at which we reach a recurrent state for the first time? Problem 31.* Absorption probabilities. Consider a Markov chain where each

state is either transient or absorbing. Fix an absorbing state s. Show that the prob

abilities ai of eventually reaching s starting from a state i are the unique solution to

the equations

as = 1,

ai = 0,

for all absorbing i t= s,

ai = L Pijaj, for all transient i.

j=l

Hint: If there are two solutions, find a system of equations that is satisfied by their difference, and look for its solutions.

Solution. The fact that the at satisfy these equations was established in the text, using the total probability theorem. To show uniqueness, let ai be another solution, and let

t5i = ai - ai. Denoting by A the set of absorbing states and using the fact t5j = 0 for

all j E A, we have

t5i = L Pijt5j = L Pijt5j, for all transient i.

j=l

j�A

Applying this relation m successive times, we obtain

t5i = L P1it L Pith ' " L Pjm-ljm . t5jm ·

it �A h�A

jm�A

Hence

= P (XI � A , . . . , Xm � A I Xo = i) . lt5jm I

=:; P (XI � A . . . . , Xm � A I Xo = i) . max lt5jl·

jltA

398 Markov Chains Chap. 7

The above relation holds for all transient i, so we obtain

f3 = P (X I � A, . . . , Xm � A I Xo = i).

Note that f3 < 1 , because there is positive probability that Xm is absorbing, regardless

of the initial state. It follows that maXj�A 18j I = 0, or ai = at for all i that are not absorbing. We also have aj = aj for all absorbing j, so ai = ai for all i.

Problem 32.* Multiple recurrent classes. Consider a Markov chain that has more that one recurrent class, as well as some transient states. Assume that all the recurrent classes are aperiodic.

( b ) Find the recurrent and the transient states. (c ) Find the expected number of years for the professor to achieve the highest ranking