An Introduction to Stochastic Optimal Control

  ❆♥ ■♥tr♦❞✉❝t✐♦♥ t♦ ❙t♦❝❤❛st✐❝ ❖♣t✐♠❛❧ ❈♦♥tr♦❧

  ❍❡rr② P✳ ❙✉r②❛✇❛♥ ❉❡♣t✳ ♦❢ ▼❛t❤❡♠❛t✐❝s✱ ❙❛♥❛t❛ ❉❤❛r♠❛ ❯♥✐✈❡rs✐t②✱ ❨♦❣②❛❦❛rt❛

  ❏✉♥❡ ✸✱ ✷✵✶✺

  ❖✉t❧✐♥❡

  ▼♦t✐✈❛t✐♦♥ ❙t♦❝❤❛st✐❝ ❖♣t✐♠❛❧ ❈♦♥tr♦❧ ❙♦♠❡ ❊①❛♠♣❧❡s ❖✉t❧♦♦❦✿ ❇❡②♦♥❞ ❇r♦✇♥✐❛♥ ▼♦t✐♦♥

  ▼♦t✐✈❛t✐♦♥

  ❚r❛❝❦✐♥❣ ❛ ❉✐✛✉s✐♦♥ P❛rt✐❝❧❡ ❯♥❞❡r ❛ ▼✐❝r♦s❝♦♣❡

❙✐t✉❛t✐♦♥ ✿ ✇❡ st✉❞② s♦♠❡ ❦✐♥❞ ♦❢ ♣❛rt✐❝❧❡s ✐♥ ❞❡t❛✐❧ ❜② ③♦♦♠✐♥❣ ✐♥ ♦♥ ♦♥❡ ♦❢

  t❤❡ ♣❛rt✐❝❧❡s✱ ✐✳❡✳ ✇❡ ✐♥❝r❡❛s❡ t❤❡ ♠❛❣♥✐✜❝❛t✐♦♥ ♦❢ t❤❡ ♠✐❝r♦s❝♦♣❡ ✉♥t✐❧ ♦♥❡ ♣❛rt✐❝❧❡ ✜❧❧s ❛ ❧❛r❣❡ ♣❛rt ♦❢ t❤❡ ✜❡❧❞ ♦❢ ✈✐❡✇✳

  

Pr♦❜❧❡♠ ✿ t❤❡ r❛♥❞♦♠ ♠♦✈❡♠❡♥t ♦❢ t❤❡ ♣❛rt✐❝❧❡ ❝❛✉s❡s ✐t t♦ r❛♣✐❞❧② ❧❡❛✈❡ ♦✉r

  ✜❡❧❞ ♦❢ ✈✐❡✇✳ ❙♦✱ ✇❡ ❤❛✈❡ t♦ ❦❡❡♣ ♠♦✈✐♥❣ ❛r♦✉♥❞ t❤❡ ❝♦✈❡r s❧✐❞❡ ✐♥ ♦r❞❡r t♦ tr❛❝❦ t❤❡ ♠♦t✐♦♥ ♦❢ t❤❡ ♣❛rt✐❝❧❡✳

  

❍♦✇ t♦ ❞♦ ✿ ✇❡ ❛tt❛❝❤ ❛♥ ❡❧❡❝tr✐❝ ♠♦t♦r t♦ t❤❡ ♠✐❝r♦s❝♦♣❡ s❧✐❞❡ ✇❤✐❝❤ ❛❧❧♦✇s

  ✉s t♦ ♠♦✈❡ t❤❡ s❧✐❞❡ ❛r♦✉♥❞✳ _t

  ▼♦❞❡❧ ✿ ▲❡t z ❜❡ t❤❡ ♣♦s✐t✐♦♥ ♦❢ t❤❡ s❧✐❞❡ r❡❧❛t✐✈❡ t♦ t❤❡ ❢♦❝✉s ♦❢ t❤❡

  ♠✐❝r♦s❝♦♣❡✳ ❚❤❡♥✱ ✇❡ ❝❛♥ ✇r✐t❡ dz _t = βu t , _t dt

  ✇❤❡r❡ u ✐s t❤❡ ✈♦❧t❛❣❡ ❛♣♣❧✐❡❞ t♦ t❤❡ ♠♦t♦r ❛♥❞ β > ✵ ✐s ❛ ❣❛✐♥ ❝♦♥st❛♥t✳ ❚❤❡ ♣♦s✐t✐♦♥ ♦❢ t❤❡ ♣❛rt✐❝❧❡ r❡❧❛t✐✈❡ t♦ t❤❡ s❧✐❞❡ ✐s ♠♦❞❡❧❧❡❞ ❜② ❛ ❇r♦✇♥✐❛♥ _t ♠♦t✐♦♥ x ✳ ❍❡♥❝❡✱ t❤❡ ♣♦s✐t✐♦♥ ♦❢ t❤❡ ♣❛rt✐❝❧❡ r❡❧❛t✐✈❡ t♦ t❤❡ ♠✐❝r♦s❝♦♣❡ _{t + z t} ❢♦❝✉s ✐s x ✳

  

●♦❛❧ ✿ t♦ ❝♦♥tr♦❧ t❤❡ s❧✐❞❡ ♣♦s✐t✐♦♥ t♦ ❦❡❡♣ t❤❡ ♣❛rt✐❝❧❡ ✐♥ ❢♦❝✉s✱ ✐✳❡✳ t♦ ❝❤♦♦s❡

  u + z _{t ✐♥ ♦r❞❡r t❤❛t x t t st❛②s ❝❧♦s❡ t♦ ③❡r♦✳}

  

✿ tr❛❝❦✐♥❣ ❛ ❇r♦✇♥✐❛♥ ♣❛rt✐❝❧❡

  ❋♦r♠❛❧✐③❛t✐♦♥ ♦❢ t❤❡ Pr♦❜❧❡♠ ❈♦st ❋✉♥❝t✐♦♥❛❧ ✿ _{Z T Z T ! !}

  ✶ ✶

  ✷ ✷

  J dt u dt _{T [u] = pE (x t + z t ) + qE , t} T T

  ✵ ✵

  ✇❤❡r❡ p ❛♥❞ q ❛r❡ ♣♦s✐t✐✈❡ ❝♦♥st❛♥ts✳

  ❋✐rst t❡r♠ ✿ t✐♠❡✲❛✈❡r❛❣❡ ♦❢ ♠❡❛♥ sq✉❛r❡ ❞✐st❛♥❝❡ ♦❢ t❤❡ ♣❛rt✐❝❧❡ ♦♥ s♦♠❡

  t✐♠❡ ✐♥t❡r✈❛❧ [✵, T ] ❢r♦♠ t❤❡ ❢♦❝✉s ♦❢ t❤❡ ♠✐❝r♦s❝♦♣❡✳

  ❙❡❝♦♥❞ t❡r♠ ✿ ❛✈❡r❛❣❡ ♣♦✇❡r ✐♥ t❤❡ ❝♦♥tr♦❧ s✐❣♥❛❧✳ ❆✐♠ ✿ t♦ ♠✐♥✐♠✐③❡ t❤❡ ✜rst ❛♥❞ s❡❝♦♥❞ t❡r♠✳ _t ❙t♦❝❤❛st✐❝ ❖♣t✐♠❛❧ ❈♦♥tr♦❧ Pr♦❜❧❡♠ ✿ t♦ ✜♥❞ t❤❡ ❢❡❡❞❜❛❝❦ str❛t❡❣② u

_{T [u]}

  ✇❤✐❝❤ ♠✐♥✐♠✐③❡ t❤❡ ❝♦st ❢✉♥❝t✐♦♥❛❧ J ✳

  ❱❛r✐❛t✐♦♥ ✿ ♣r♦❜❧❡♠ ♦♥ ✐♥✜♥✐t❡ t✐♠❡ ❤♦r✐③♦♥ [✵, ∞)✿ _{Z T Z T ! !}

  ✶ ✶

_{E E}

✷ ✷ J dt u dt _{∞ [u] = p (x t + z t ) + q .}

  ❧✐♠ s✉♣ ❧✐♠ s✉♣ t _{T →∞ T →∞} T T

  ✵ ✵

  ❙t♦❝❤❛st✐❝ ❖♣t✐♠❛❧ ❈♦♥tr♦❧

  ❆ ❇r✐❡❢ ❍✐st♦r②

■♠♣❡t✉s ♦❢ ❖♣t✐♠❛❧ ❈♦♥tr♦❧ ✿ ❈❛❧❝✉❧✉s ♦❢ ❱❛r✐❛t✐♦♥ ✭✐♥ ✶✻✻✷ P✐❡rr❡ ❞❡ ❋❡r♠❛t

  ✉s❡❞ t❤❡ ♠❡t❤♦❞ ♦❢ ❝❛❧❝✉❧✉s t♦ ♠✐♥✐♠✐③❡ t❤❡ ♣❛ss❛❣❡ t✐♠❡ ♦❢ ❛ ❧✐❣❤t r❛② t❤r♦✉❣❤ t✇♦ ♦♣t✐❝❛❧ ♠❡❞✐❛✮ ❛♥❞ ❇r❛❝❤✐st♦❝❤r♦♥❡ ♣r♦❜❧❡♠ ✭✐♥ ✶✻✾✻ ❜② ❏♦❤❛♥ ❇❡r♥♦✉❧❧✐✮✳

  ❊❛r❧② ✇♦r❦ ♦♥ ♦♣t✐♠❛❧ ❝♦♥tr♦❧ t❤❡♦r② ✿ ✐♥ ✶✽✵✵s✱ ❡✳❣✳ ❍❛♠✐❧t♦♥✱ ❍✉r✇✐t③✱

  ▼❛①✇❡❧❧✱ P♦✐♥❝❛r❡✱ ▲②❛♣✉♥♦✈✱ ❲✐❡♥❡r✱ ❑♦❧♠♦❣♦r♦❢

  

▼♦❞❡r♥ ♦♣t✐♠❛❧ ❝♦♥tr♦❧ t❤❡♦r② ✿ ✐♥ t❤❡ ❡♥❞ ♦❢ t❤❡ ❲♦r❧❞ ❲❛r ■■✱ ❡✳❣✳ ❇❡❧❧♠❛♥✱

  ▲❛❙❛❧❧❡✱ ❇❧❛❝❦✇❡❧❧✱ ❋❧❡♠✐♥❣✱ ❇❡r❦♦✈✐t③✱ P♦♥tr②❛❣✐♥✱ ❑❛❧♠❛♥✳

  

❇❡❧❧♠❛♥✬s ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ♠❡t❤♦❞ ✱ P♦♥tr②❛❣✐♥✬s ♠❛①✐♠✉♠ ♣r✐♥❝✐♣❧❡ ✱

❑❛❧♠❛♥✬s ▲◗ t❤❡♦r② ❙t♦❝❤❛st✐❝ ♦♣t✐♠❛❧ ❝♦♥tr♦❧ ✿

  ❇❡❧❧♠❛♥ ✐♥ ✶✾✺✷ ♠❡♥t✐♦♥❡❞ ✑st♦❝❤❛st✐❝ ❝♦♥tr♦❧✑ ✐♥ ♦♥❡ ♦❢ ❤✐s ❡❛r❧✐❡st ♣❛♣❡r✱ ❜✉t ■t♦✲t②♣❡ ❙❉❊ ✇❛s ♥♦t ✐♥✈♦❧✈❡❞✦ ❋❧♦r❡♥t✐♥ ✐♥ ✶✾✻✶ ❞❡r✐✈❡❞ ❢r♦♠ ❛ P❉❊ ❛ss♦❝✐❛t❡❞ ✇✐t❤ ❛ ❝♦♥tr♦❧❧❡❞ ▼❛r❦♦✈ ♣r♦❝❡ss ❜② ✉s✐♥❣ ❇❡❧❧♠❛♥✬s ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣✳ ❑✉s❤♥❡r ✐♥ ✶✾✻✷ st✉❞✐❡❞ ♦♣t✐♠❛❧ ❝♦♥tr♦❧ ♣r♦❜❧❡♠ ✉s✐♥❣ ■t♦✲t②♣❡ ❙❉❊ ❛s t❤❡ st❛t❡ ❡q✉❛t✐♦♥✳ ▼❡rt♦♥ ✐♥ ❧❛t❡ ✶✾✻✵s✿ ✜rst ❛♣♣❧✐❝❛t✐♦♥ ♦❢ st♦❝❤❛st✐❝ ♦♣t✐♠❛❧ ❝♦♥tr♦❧ t❤❡♦r② t♦ ✜♥❛♥❝✐❛❧ ♠❛t❤❡♠❛t✐❝s✳

  ❙t♦❝❤❛st✐❝ ❖♣t✐♠❛❧ ❈♦♥tr♦❧

❙t♦❝❤❛st✐❝ ❖♣t✐♠❛❧ ❈♦♥tr♦❧ Pr♦❜❧❡♠ ✭❙❖❈P✮ ✿ ❝♦♠♣❧❡t❡❧② ♦❜s❡r✈❡❞ ❝♦♥tr♦❧

  ♣r♦❜❧❡♠ ✇✐t❤ ❛ st❛t❡ ❡q✉❛t✐♦♥ ♦❢ t❤❡ ■t♦ t②♣❡ ✭❞✐✛✉s✐♦♥ ♠♦❞❡❧✮ ❛♥❞ ✇✐t❤ ❛ ❝♦st ❢✉♥❝t✐♦♥❛❧ ♦❢ t❤❡ ❇♦❧③❛ t②♣❡✳ ❚❤❡ ❜❛s✐❝ s♦✉r❝❡ ♦❢ ✉♥❝❡rt❛✐♥t② ✐♥ ❞✐✛✉s✐♦♥ ♠♦❞❡❧s ✐s ✇❤✐t❡ ♥♦✐s❡ ✭t❤❡ t✐♠❡ ✑❞❡r✐✈❛t✐✈❡✑ ♦❢ t❤❡ ❇r♦✇♥✐❛♥ ♠♦t✐♦♥✮✱ ✇❤✐❝❤ r❡♣r❡s❡♥ts t❤❡ ❥♦✐♥t ❡✛❡❝ts ♦❢ ❛ ❧❛r❣❡ ♥✉♠❜❡r ♦❢ ✐♥❞❡♣❡♥❞❡♥t r❛♥❞♦♠ ❢♦r❝❡s ❛❝t✐♥❣ ♦♥ t❤❡ s②st❡♠s✳ ❚✇♦ ♠❛✐♥ ❛♣♣r♦❛❝❤❡s✿ _✶

  P♦♥tr②❛❣✐♥✬s ♠❛①✐♠✉♠ ♣r✐♥❝✐♣❧❡ ✿ ❍❛♠✐❧t♦♥✐❛♥ s②st❡♠ ✇❤✐❝❤ ❝♦♥s✐sts ♦❢ ❛❞❥♦✐♥t _✷

❡q✉❛t✐♦♥ ✭❖❉❊✴❙❉❊✮✱ ♦r✐❣✐♥❛❧ st❛t❡ ❡q✉❛t✐♦♥ ❛♥❞ ♠❛①✐♠✉♠ ❝♦♥❞✐t✐♦♥s

❇❡❧❧♠❛♥✬s ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ✿ s♦❧✈✐♥❣ ❍❏❇ ❡q✉❛t✐♦♥ ✇❤✐❝❤ ✐s ❛ P❉❊ ♦❢

✜rst ♦r❞❡r ✐♥ ❞❡t❡r♠✐♥✐st✐❝ ❝❛s❡ ❛♥❞ ♦❢ s❡❝♦♥❞ ♦r❞❡r ✐♥ st♦❝❤❛st✐❝ ❝❛s❡✳

  ❘❡❢❡r❡♥❝❡s ✿ _✶ ❲✳ ❍✳ ❋❧❡♠✐♥❣ ❛♥❞ ❍✳ ▼✳ ❙♦♥❡r✳ ✷✵✵✻✳ ✑❈♦♥tr♦❧❧❡❞ ▼❛r❦♦✈ Pr♦❝❡ss❡s ❛♥❞ _✷ ❱✐s❝♦s✐t② ❙♦❧✉t✐♦♥s✑✳ ❙♣r✐♥❣❡r✳ ❏✳ ❨♦♥❣ ❛♥❞ ❳✳ ❨✳ ❩❤♦✉✳ ✷✵✵✾✳ ✑❙t♦❝❤❛st✐❝ ❈♦♥tr♦❧s✳ ❍❛♠✐❧t♦♥✐❛♥ ❙②st❡♠s _✸ ❛♥❞ ❍❏❇ ❊q✉❛t✐♦♥s✑✳ ❙♣r✐♥❣❡r✳ ▼✳ ◆✐s✐♦✳ ✷✵✶✺✳ ✑❙t♦❝❤❛st✐❝ ❈♦♥tr♦❧ ❚❤❡♦r②✳ ❉②♥❛♠✐❝ Pr♦❣r❛♠♠✐♥❣

  Pr✐♥❝✐♣❧❡✑✳ ❙♣r✐♥❣❡r✳

  ❇❛s✐❝s ♦❢ ❙❖❈P

  ) , P ▲❡t Ω, F, (F t ❜❡ ❛ ✜❧t❡r❡❞ ♣r♦❜❛❜✐❧✐t② s♣❛❝❡ ♦♥ ✇❤✐❝❤ ❛♥ F t ✲❛❞❛♣t❡❞ _t≥

  ✵

  ) ❇r♦✇♥✐❛♥ ♠♦t✐♦♥ B = (B t t≥ ✐s ❞❡✜♥❡❞✳ ❚❤❡ st❛t❡ ♦❢ t❤❡ ❝♦♥tr♦❧❧❡❞ s②st❡♠ ✐♥ ❛♥

  ✵

  ❙❖❈P ✐s ❞❡s❝r✐❜❡❞ ❜② ❛♥ ❙❉❊ _{u u u} dX = b (t, X , u t ) dt + σ (t, X , u t ) dB t , _{t t t} X = x.

_{n n n n×m}

✵ ❍❡r❡ b : [✵, ∞) × R × U → R ❛♥❞ σ : [✵, ∞) × R × U → R ✱ ✇❤❡r❡ U ✐s t❤❡ _q ❝♦♥tr♦❧ s❡t ✭t❤❡ s❡t ♦❢ ✈❛❧✉❡s t❤❛t t❤❡ ❝♦♥tr♦❧ ✐♥♣✉t ❝❛♥ t❛❦❡✮✳ ❖❢t❡♥✿ U = R ✳

  ❉❡✜♥✐t✐♦♥ _{t ) t≥}

  ❚❤❡ ❝♦♥tr♦❧ str❛t❡❣② u = (u ✵ ✐s ❝❛❧❧❡❞ ❛♥ ❛❞♠✐ss✐❜❧❡ str❛t❡❣② ✐❢ _✶ u _{t t ✷} ✐s ❛♥ F ✲❛❞❛♣t❡❞ st♦❝❤❛st✐❝ ♣r♦❝❡ss u _{t ✸} (ω) ∈ U ❢♦r ❡✈❡r② (ω, t) ∈ Ω × [✵, ∞) _u t❤❡ ❡q✉❛t✐♦♥ ❢♦r X t ❤❛s ❛ ✉♥✐q✉❡ str♦♥❣ s♦❧✉t✐♦♥✳ ▼♦r❡♦✈❡r✱ ❛♥ ❛❞♠✐ss✐❜❧❡ str❛t❡❣② u ✐s ❝❛❧❧❡❞ ❛ ▼❛r❦♦✈ str❛t❡❣② ✐❢ ✐t ✐s ♦❢ t❤❡ ❢♦r♠ _{u n} u _{t = α (t, X ) t ❢♦r s♦♠❡ ❢✉♥❝t✐♦♥ α : [✵, ∞) × R → U✳}

  ❚❤r❡❡ ❝♦♠♠♦♥ t②♣❡s ♦❢ ❝♦st ❢✉♥❝t✐♦♥❛❧s✿ _✶

  ❙❖❈P ♦♥ t❤❡ ✜♥✐t❡ t✐♠❡ ❤♦r✐③♦♥ [✵, T ] ✿ _{T !}

  Z _{u u} J w

  [u] = E (s, X , u s ) ds + z (X ) , _{s T n n} ✵ ✇❤❡r❡ w : [✵, T ] × R × U → R ✭t❤❡ r✉♥♥✐♥❣ ❝♦st✮ ❛♥❞ z : R → R ✭t❤❡ _✷ t❡r♠✐♥❛❧ ❝♦st✮ ❛r❡ ♠❡❛s✉r❛❜❧❡ ❢✉♥❝t✐♦♥s ❛♥❞ T < ∞ ✐s t❤❡ t❡r♠✐♥❛❧ t✐♠❡✳

  ❙❖❈P ♦♥ ❛♥ ✐♥❞❡✜♥✐t❡ t✐♠❡ ❤♦r✐③♦♥ ✿ _{τ u !}

  Z _{u u u} J w

  [u] = E (X , u s ) ds + z (X ) , _{s τ n} ✵ ✇❤❡r❡ S ⊂ R ✱ w : S × U → R✱ z : ∂S → R✱ ❛♥❞ t❤❡ st♦♣♣✐♥❣ t✐♠❡ _{u u} τ = / _✸ ✐♥❢ {t > ✵ : X t ∈ S}

  ❙❖❈P ♦♥ ❛♥ ✐♥✜♥✐t❡ t✐♠❡ ❤♦r✐③♦♥ ✿ _{Z T ! E} ✶ u u _u

  J w [u] = (X , u s ) ds + z (X ) ,

  ❧✐♠ s✉♣ s τ _{T →∞} T _n ✵ ✇❤❡r❡ w : R × U → R ✐s ♠❡❛s✉r❛❜❧❡✳

  ❇❡❧❧♠❛♥✬s ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ♠❡t❤♦❞

  ●✐✈❡♥ t❤❡ st❛t❡ ♦❢ t❤❡ ❝♦♥tr♦❧❧❡❞ s②st❡♠ _{u u u} dX

  X _{t t t ✵} = b (t, X , u t ) dt + σ (t, X , u t ) dB t , = x ✇✐t❤ ❛ ❝♦st ❢✉♥❝t✐♦♥❛❧ J[u]✳

  ❚❤❡ ❣♦❛❧ ✐s t♦ ✜♥❞ _∗

  u = _α ♠✐♥ {J[u] : u ∈ U ✐s ❛ ▼❛r❦♦✈ str❛t❡❣② } ❉❡✜♥❡ t❤❡ ❣❡♥❡r❛t♦r L t ✱ α ∈ U ❛s _{n n m X X X} ✷ _{α ✶ i ik (t,x,α) jk} ∂g ∂ g g b

  (x) = (t, x, α) (x) + σ σ (t, x, α) (x) L t _{i i j}

  ∂x ∂x ∂x _{i = i ,j = k =} ✷

  ✶ ✶ ✶ _u

  J ❛♥❞ t❤❡ ❝♦st✲t♦✲❣♦ ❢✉♥❝t✐♦♥ t ♦❢ t❤❡ ▼❛r❦♦✈ str❛t❡❣② u ❛s _{T ! u u u u} Z

  J w _{t t s T F} (X ) = E (s, X , u s ) ds + z (X ) t . _t

  ❚❤❡♦r❡♠ _{t (x)} ✶ ✷

  ❙✉♣♣♦s❡ t❤❡r❡ ✐s ❛ V ✱ ✇❤✐❝❤ ✐s C ✐♥ t ❛♥❞ C ✐♥ x✱ s✉❝❤ t❤❛t ∂V t (x) ^α

• ∂t

  V V (x) = z(x), ♠✐♥ {L t (x) + w (t, x, α)} = ✵, T _{α∈U t}

  (X ❛♥❞ |E (V ✵ ✵ )))| < ∞✱ ❛♥❞ ❝❤♦♦s❡ ❛ ♠✐♥✐♠✉♠ _{∗ α}

  V α t (t, x) ∈ ❛r❣♠✐♥ α∈U {L t (x) + w (t, x, α)} .

  ❉❡♥♦t❡ ❜② K t❤❡ ❝❧❛ss ♦❢ ❛❞♠✐ss✐❜❧❡ str❛t❡❣✐❡s u s✉❝❤ t❤❛t _{X n m X Z t} ∂V _{s u ik u k i} (X ) σ (s, X , u s ) dB _{s s s}

  ∂x _{i = k =} ✵ ✶ ✶ _{∗ ∗ u ∗} t

  (t) = α , X ✐s ❛ ♠❛rt✐♥❣❛❧❡✱ ❛♥❞ s✉♣♣♦s❡ t❤❛t t❤❡ ❝♦♥tr♦❧ u t ❞❡✜♥❡s ❛♥ _∗ ❛❞♠✐ss✐❜❧❡ ▼❛r❦♦✈ str❛t❡❣② ✇❤✐❝❤ ✐s ✐♥ K✳ ❚❤❡♥✱ J[u ] ≤ J[u] ❢♦r ❛♥② u ∈ K✱ ❛♥❞ _{u ∗}

  V _{t (x) = J (x) t ✐s t❤❡ ✈❛❧✉❡ ❢✉♥❝t✐♦♥ ❢♦r t❤❡ ❝♦♥tr♦❧ ♣r♦❜❧❡♠✳}

  ❙♦♠❡ ❊①❛♠♣❧❡s

  ❊①❛♠♣❧❡ ✶✿ ❚r❛❝❦✐♥❣ ❛ P❛rt✐❝❧❡

  ❙②st❡♠✿ dz _t

  = βu , x = x + σB , _{t t t}

  ✵

  dt z _t ✐s t❤❡ ♣♦s✐t✐♦♥ ♦❢ t❤❡ s❧✐❞❡ r❡❧❛t✐✈❡ t♦ t❤❡ ❢♦❝✉s ♦❢ t❤❡ ♠✐❝r♦s❝♦♣❡ x _t

  ✐s t❤❡ ♣♦s✐t✐♦♥ ♦❢ t❤❡ ♣❛rt✐❝❧❡ ✇❡ ✇✐s❤ t♦ ✈✐❡✇ ✉♥❞❡r t❤❡ ♠✐❝r♦s❝♦♣❡ r❡❧❛t✐✈❡ t♦ t❤❡ ❝❡♥t❡r ♦❢ t❤❡ s❧✐❞❡ β ∈ R ✐s t❤❡ ❣❛✐♥ ❝♦♥st❛♥t σ >

  ✵ ✐s t❤❡ ❞✐✛✉s✐♦♥ ❝♦♥st❛♥t ♦❢ t❤❡ ♣❛rt✐❝❧❡ _{t + z t} ❆✐♠✿ t♦ ❦❡❡♣ t❤❡ ♣❛rt✐❝❧❡ ✐♥ ❢♦❝✉s ✭❦❡❡♣ x ❛s ❝❧♦s❡ t♦ ③❡r♦ ❛s ♣♦ss✐❜❧❡✮ ❛♥❞ t♦ ✐♥tr♦❞✉❝❡ ❛ ♣♦✇❡r ❝♦♥str❛✐♥t ♦♥ t❤❡ ❝♦♥tr♦❧ ❛s ✇❡❧❧✱ ❛s ✇❡ ❝❛♥♥♦t ❞r✐✈❡ t❤❡ s❡r✈♦ ♠♦t♦r ✇✐t❤ ❛r❜✐tr❛r✐❧② ❧❛r❣❡ ✐♥♣✉t ♣♦✇❡rs✳ ❈♦st ❋✉♥❝t✐♦♥❛❧✿ _{Z T Z T !} p q

  ✷ ✷

  J dt dt [u] = E (x t + z t ) (u t ) , +

  T T

  

✵ ✵

  ✇❤❡r❡ p, q > ✵ ❛❧❧♦✇ ✉s t♦ s❡❧❡❝t t❤❡ tr❛❞❡♦✛ ❜❡t✇❡❡♥ ❣♦♦❞ tr❛❝❦✐♥❣ ❛♥❞ ❧♦✇ ❢❡❡❞❜❛❝❦ ♣♦✇❡r✳

  ❊①❛♠♣❧❡ ✶✿ ❚r❛❝❦✐♥❣ ❛ P❛rt✐❝❧❡✳ ❝♦♥t✬❞✳

_{t + z t}

  ❆s t❤❡ ❝♦♥tr♦❧ ❝♦st ❞❡♣❡♥❞s ♦♥❧② ♦♥ x ✱ ✐t ✐s ♠♦r❡ ❝♦♥✈❡♥✐❡♥t t♦ ♣r♦❝❡❡❞ _{t = x t + z t p q} ❞✐r❡❝t❧② ✇✐t❤ t❤✐s q✉❛♥t✐t②✳ ❚❤❛t ✐s✱ ✇❡ ❞❡✜♥❡ e ✱ P = ❞❛♥ Q = _{T T} ❛♥❞ ♥♦t❡ t❤❛t _{Z T Z T !}

  ✷ ✷

  de t = βu t dt + σ dB t , J [u] = E P (x t + z t ) dt + Q (u t ) dt

  ✵ ✵

  ❲❡ ♦❜t❛✐♥ t❤❡ ❍❏❇ ❡q✉❛t✐♦♥✿

  ✷ ✷

  V ∂V t (x) σ ∂ t (x) ∂V t (x)

  ✷ ✷

• _α∈R

  ✷

• β α + Px + Qα ✵ = ♠✐♥

  ∂t ∂x ∂x ✷

  ✷ ✷ ✷ ✷

  V ∂V t (x) σ ∂ t (x) β ∂V t (x)

  ✷

  = + + Px −

  ✷

  ∂t ∂x ∂x _{T (x) =} ✷ ✹Q ✇✐t❤ V ✵ ✭❛s t❤❡r❡ ✐s ♥♦ t❡r♠✐♥❛❧ ❝♦st✮✱ ❛♥❞ ♠♦r❡♦✈❡r _∗ ∂V t (x) β ∂V t (x)

  ✷ α (t, x) = β α + Qα .

  ❛r❣♠✐♥ α∈R = − ∂x ∂x

  ✷Q

  ❊①❛♠♣❧❡ ✶✿ ❚r❛❝❦✐♥❣ ❛ P❛rt✐❝❧❡✳ ❝♦♥t✬❞✳ ✷ _{t (x) = a t + b t T (x) =} x

  ❯s❡ ❛♥s❛t③ V ❛♥❞ V ✵✿

  ✷

  da t β db t

  ✷ ✷

  a a = T = + σ t = T = + P − t ✵, a ✵, ✵, b ✵. dt Q dt

  ❙♦❧✉t✐♦♥✿ _{r r ! !!} √

  ✷

  PQ P Q P σ a b _{t = β , t = β} t❛♥❤ (T − t) ❧♥ ❝♦s❤ (T − t)

  ✷

  β Q β Q _{t (x) (t, x) ∗} ◆♦t❡ t❤❛t V ✐s s♠♦♦t❤ ✐♥ x ❛♥❞ t ❛♥❞ t❤❛t α ✐s ✉♥✐❢♦r♠❧② ▲✐♣s❝❤✐t③ ♦♥

  ✷

  [ + z ) ✵, T ]✳ ❍❡♥❝❡ ✐❢ ✇❡ ❛ss✉♠❡ t❤❛t E (x t t < ∞✱ t❤❡♥ ✇❡ ✜♥❞ t❤❛t t❤❡

  ❢❡❡❞❜❛❝❦ ❝♦♥tr♦❧ _{r r ! ∗ ∗} P P u _{t ) = − t❛♥❤ (T − t)} = α (t, e t β (x t + z t ) _{∗ ∗} Q Q s❛t✐s✜❡s u ∈ K✳ ❚❤✉s✱ ❜② t❤❡ ❛❜♦✈❡ t❤❡♦r❡♠ u ✐s ❛♥ ♦♣t✐♠❛❧ ❝♦♥tr♦❧ str❛t❡❣②✳ _{t t}

  ❊①❛♠♣❧❡ ✷✿ ❖♣t✐♠❛❧ P♦rt❢♦❧✐♦ ❙❡❧❡❝t✐♦♥

  ❲❡ ❝♦♥s✐❞❡r✿ ❛ s✐♥❣❧❡ st♦❝❦ ✇✐t❤ ❛✈❡r❛❣❡ r❡t✉r♥ µ > ✵ ❛♥❞ ✈♦❧❛t✐❧✐t② σ > ✵ ❛ ❜❛♥❦ ❛❝❝♦✉♥t ✇✐t❤ ✐♥t❡r❡st r❛t❡ r > ✵✳

  ❉②♥❛♠✐❝ ♦❢ t❤❡ s②st❡♠ ✿

  dS = µS dt + σS dB , S = = rR dt , R = _{t t t t ✶, dR t t ✶.}

  ✵ ✵ ❆ss✉♠♣t✐♦♥ ✿ _{✶ ✷} ❲❡ ❝❛♥ ♠♦❞✐❢② ♦✉r ✐♥✈❡st♠❡♥t ❛t ❛♥② ♣♦✐♥t ✐♥ t✐♠❡✳

  ❲❡ ❝♦♥s✐❞❡r s❡❧❢✲✜♥❛♥❝✐♥❣ ✐♥✈❡st♠❡♥t str❛t❡❣✐❡s _{t t} ▲❡t X ❜❡ t❤❡ t♦t❛❧ ✇❡❛❧t❤ ❛t t✐♠❡ t✱ ❛♥❞ ❜② u t❤❡ ❢r❛❝t✐♦♥ ♦❢ ♦✉r ✇❡❛❧t❤ t❤❛t ✐s ✐♥✈❡st❡❞ ✐♥ st♦❝❦ ❛t t✐♠❡ t✳ ❚❤❡♥ t❤❡ s❡❧❢✲✜♥❛♥❝✐♥❣ ❝♦♥❞✐t✐♦♥ ✐♠♣❧✐❡s t❤❛t dX dt _{t t + r ( t t + σu t t t .} X dB

  = {µu ✶ − u )} X

  ●♦❛❧ ✿ ✭♦❜✈✐♦✉s❧②✮ t♦ ♠❛❦❡ ♠♦♥❡②✦

  ❊①❛♠♣❧❡ ✷✿ ❖♣t✐♠❛❧ P♦rt❢♦❧✐♦ ❙❡❧❡❝t✐♦♥✳ ❝♦♥t✬❞✳ _t

  ▲❡t ✜① ❛ t❡r♠✐♥❛❧ t✐♠❡ T ✱ ❛♥❞ tr② t♦ ❝❤♦♦s❡ ❛ str❛t❡❣② u t❤❛t ♠❛①✐♠✐③❡s ❛ s✉✐t❛❜❧❡ ❢✉♥❝t✐♦♥❛❧ U ♦❢ ♦✉r t♦t❛❧ ✇❡❛❧t❤ ❛t t✐♠❡ T ❀ ✐✳❡✳ ✱ ✇❡ ❝❤♦♦s❡ t❤❡ ❝♦st ❢✉♥❝t✐♦♥❛❧ _u

  J )) .

  [u] = E (−U (X T ❍♦✇ t♦ ❝❤♦♦s❡ ✉t✐❧✐t② ❢✉♥❝t✐♦♥❄

  ❚❤❡ ♦❜✈✐♦✉s ❝❤♦✐❝❡ U(x) = x t✉r♥s ♦✉t ♥♦t t♦ ❛❞♠✐t ❛♥ ♦♣t✐♠❛❧ ❝♦♥tr♦❧ ✐❢ ✇❡ s❡t U = R✱ ✇❤✐❧❡ ✐❢ ✇❡ s❡t U = [✵, ✶] ✭✇❡ ❞♦ ♥♦t ❛❧❧♦✇ ❜♦rr♦✇✐♥❣ ♠♦♥❡② ♦r s❡❧❧✐♥❣ s❤♦rt✮ t❤❡♥ ✇❡ ❣❡t ❛ r❛t❤❡r ❜♦r✐♥❣ ❛♥s✇❡r✿ ✇❡ s❤♦✉❧❞ ❛❧✇❛②s ♣✉t ❛❧❧ ♦✉r ♠♦♥❡② ✐♥ st♦❝❦ ✐❢ µ > r✱ ✇❤✐❧❡ ✐❢ µ ≤ r ✇❡ s❤♦✉❧❞ ♣✉t ❛❧❧ ♦✉r ♠♦♥❡② ✐♥ t❤❡ ❜❛♥❦✳ ❙✉♣♣♦s❡ t❤❛t U ✐s ♥♦♥❞❡❝r❡❛s✐♥❣ ❛♥❞ ❝♦♥❝❛✈❡✱ ❡✳❣✳✱ U(x) = ❧♥(x)✳ ❚❤❡♥ t❤❡ r❡❧❛t✐✈❡ ♣❡♥❛❧t② ❢♦r ❡♥❞✐♥❣ ✉♣ ✇✐t❤ ❛ ❧♦✇ t♦t❛❧ ✇❡❛❧t❤ ✐s ♠✉❝❤ ❤❡❛✈✐❡r t❤❛♥ ❢♦r U(x) = x✱ s♦ t❤❛t t❤❡ r❡s✉❧t✐♥❣ str❛t❡❣② ✇✐❧❧ ❜❡ ❧❡ss r✐s❦② ✭❝♦♥❝❛✈❡ ✉t✐❧✐t② ❢✉♥❝t✐♦♥s ❧❡❛❞ t♦ r✐s❦✲❛✈❡rs❡ str❛t❡❣✐❡s✱ ✇❤✐❧❡ t❤❡ ✉t✐❧✐t② U(x) = x ✐s ❝❛❧❧❡❞ r✐s❦✲♥❡✉tr❛❧✮✳ ❆s s✉❝❤✱ ✇❡ ✇♦✉❧❞ ❡①♣❡❝t t❤✐s ✐❞❡❛ t♦ t❡❧❧ ✉s t♦ ♣✉t s♦♠❡ ♠♦♥❡② ✐♥ t❤❡ ❜❛♥❦ t♦ r❡❞✉❝❡ ♦✉r r✐s❦✦

  ❊①❛♠♣❧❡ ✷✿ ❖♣t✐♠❛❧ P♦rt❢♦❧✐♦ ❙❡❧❡❝t✐♦♥✳ ❝♦♥t✬❞✳

  ❚❤❡ ❇❡❧❧♠❛♥ ❡q✉❛t✐♦♥ r❡❛❞s ✭✇✐t❤ U = R✮

  ✷ ✷ ✷ ✷

  ∂V t (x) σ α x ∂ V t (x) ∂V t (x)

• _α∈R
• (µα + r ( ✵ = ♠✐♥ ✶ − α)) x

  

✷

  ∂t ∂x ∂x ✷

  ✷

✷

  ∂V t (x) ∂V t (x) (∂V t (x)/∂x) (µ − r)

  = + rx , −

  ✷ ✷ ✷

  V ∂t ∂x ∂ t (x)/∂x _T ✷σ

  ✇❤❡r❡ V (x) = − ❧♥ x✱ ❛♥❞ ♠♦r❡♦✈❡r ∂V (x)/∂x _∗ µ − r t

  α , (t, x) = −

  ✷ ✷ ✷

  x

  V σ ∂ t (x)/∂x

  ✷ ✷

  V _{t (x)/∂x >} ♣r♦✈✐❞❡❞ t❤❛t ∂ ✵ ❢♦r ❛❧❧ x > ✵✳ _{t t} ❆♥s❛t③✿ V (x) = − ❧♥ x + b ✱ ❛♥❞

  ✷

  db t (µ − r) b _{T = .} − C = ✵, ✵, C = r +

  ✷

  dt ✷σ

  ❊①❛♠♣❧❡ ✷✿ ❖♣t✐♠❛❧ P♦rt❢♦❧✐♦ ❙❡❧❡❝t✐♦♥✳ ❝♦♥t✬❞✳

  ❙♦❧✉t✐♦♥ t♦ ❇❡❧❧♠❛♥ ❡q✉❛t✐♦♥✿ V t

  (x) = − ❧♥ x − C (T − t)

  ✷ ✷

  V _{t (x)/∂x >} ✭s♠♦♦t❤ ♦♥ x > ✵ ❛♥❞ ∂ ✵✮✳ ❚❤❡ ❝♦rr❡s♣♦♥❞✐♥❣ ❝♦♥tr♦❧✿ _∗ µ − r α (t, x) = .

  ✷

  σ ❇② ❡①✐st❡♥❝❡ t❤❡♦r❡♠ ♦❢ ❙❉❊✱ t❤❡ ❝♦♥❞✐t✐♦♥s ♦❢ ❛❜♦✈❡ t❤❡♦r❡♠ ❛r❡ ♠❡t ❛♥❞ ✇❡ ✜♥❞ t❤❛t t❤❡ ♦♣t✐♠❛❧ ❝♦♥tr♦❧✿

  µ − r u _{t = .}

  ✷

  σ ❍❡♥❝❡✱ t❤❡ ❝❤♦✐❝❡ ♦❢ t❤❡ ✉t✐❧✐t② ❢✉♥❝t✐♦♥ t❡❧❧s ✉s t♦ ♣✉t ♠♦♥❡② ✐♥ t❤❡ ❜❛♥❦✱

  ✷

  ♣r♦✈✐❞❡❞ t❤❛t µ − r < σ ✳ ❖♥ t❤❡ ♦t❤❡r ❤❛♥❞✱ ✐❢ µ − r ✐s ❧❛r❣❡✱ ✐t ✐s ❜❡tt❡r t♦ ❜♦rr♦✇ ♠♦♥❡② ❢r♦♠ t❤❡ ❜❛♥❦ t♦ ✐♥✈❡st ✐♥ st♦❝❦ ✭t❤✐s ✐s ♣♦ss✐❜❧❡ ✐♥ t❤❡ ❝✉rr❡♥t s❡tt✐♥❣ ❛s ✇❡ ❤❛✈❡ ❝❤♦s❡♥ U = R✱ r❛t❤❡r t❤❛♥ r❡str✐❝t✐♥❣ t♦ U = [✵, ✶]✮✳

  ❖✉t❧♦♦❦✿ ❇❡②♦♥❞ ❇r♦✇♥✐❛♥ ▼♦t✐♦♥

  ■❞❡❛ ❢♦r ❣❡♥❡r❛❧✐③❛t✐♦♥s✴✐♠♣r♦✈❡♠❡♥ts✿ r❡♣❧❛❝❡ t❤❡ ❞r✐✈✐♥❣ ♣r♦❝❡ss✱ _t ❇r♦✇♥✐❛♥ ♠♦t✐♦♥ B ✱ ✐♥ t❤❡ st❛t❡ ♦❢ ❝♦♥tr♦❧❧❡❞ s②st❡♠ ✇✐t❤ s♦♠❡ ♦t❤❡r _t st♦❝❤❛st✐❝ ♣r♦❝❡ss M _{u u u} dX M

  X _{t t t ✵} = b (t, X , u t ) dt + σ (t, X , u t ) d t , = x _t ❈♦♠♠♦♥ ♦♣t✐♦♥s ❢♦r M ✿ _✶

  ▲é✈② ♣r♦❝❡ss ✿ ♣r♦❝❡ss ✇✐t❤ st❛t✐♦♥❛r② ❛♥❞ ✐♥❞❡♣❡♥❞❡♥t ✐♥❝r❡♠❡♥ts ✇✐t❤ ❝á❞❧á❣ s❛♠♣❧❡ ♣❛t❤s P✲❛✳s✳ ❋❡❛t✉r❡s✿ ✐♥✜♥✐t❡❧② ❞✐✈✐s✐❜❧❡ ❞✐str✐❜✉t❡❞✱ ▼❛r❦♦✈✱ s❡♠✐♠❛rt✐♥❣❛❧❡✱ ❥✉♠♣s ✐♥ t❤❡ tr❛❥❡❝t♦r✐❡s ❘❡❢❡r❡♥❝❡ ✿ ❇✳ ❖❦s❡♥❞❛❧ ❛♥❞ ❆✳ ❙✉❧❡♠✳ ✷✵✵✺✳ ✑❆♣♣❧✐❡❞ ❙t♦❝❤❛st✐❝ ❈♦♥tr♦❧ ♦❢ _✷ ❏✉♠♣ ❉✐✛✉s✐♦♥s✑✳ ❙♣r✐♥❣❡r _, ❋r❛❝t✐♦♥❛❧ ❇r♦✇♥✐❛♥ ♠♦t✐♦♥ ✇✐t❤ ❍✉rst ♣❛r❛♠❡t❡r H ∈ (✵ ✶) ✐s ❛ ❝❡♥t❡r❡❞ _{H H t≥}

  := (B t ) ●❛✉ss✐❛♥ ♣r♦❝❡ss B ✵ ✇✐t❤ ❝♦✈❛r✐❛♥❝❡ ❢✉♥❝t✐♦♥ _{H H E B t} ✶ ✷H ✷H ✷H (B t s ) = + s − |t − s| ✳

  ✷ ❋❡❛t✉r❡s✿ ♥♦r♠❛❧❧② ❞✐str✐❜✉t❡❞✱ ❝♦♥t✐♥✉♦✉s tr❛❥❡❝t♦r✐❡s✱ ❧♦♥❣✴s❤♦rt ♠❡♠♦r②✱ ♥♦♥✲▼❛r❦♦✈ ✱ ♥♦♥✲♠❛rt✐♥❣❛❧❡

❘❡❢❡r❡♥❝❡ ✿ ❋✳ ❇✐❛❣✐♥✐ ❡t✳ ❛❧✳ ✷✵✵✽✳ ✑❙t♦❝❤❛st✐❝ ❈❛❧❝✉❧✉s ❢♦r ❋r❛❝t✐♦♥❛❧

❇r♦✇♥✐❛♥ ▼♦t✐♦♥ ❛♥❞ ❆♣♣❧✐❝❛t✐♦♥s✑✳ ❙♣r✐♥❣❡r

  

✿ ❇r♦✇♥✐❛♥ ♠♦t✐♦♥

✿ ❋r❛❝t✐♦♥❛❧ ❇r♦✇♥✐❛♥ ♠♦t✐♦♥ ✇✐t❤ ❞✐✛❡r❡♥t H

  

✿ P♦✐ss♦♥ ♣r♦❝❡ss

✿ ■♥❞❡♣❡♥❞❡♥t s✉♠ ♦❢ ❝♦♠♣♦✉♥❞ P♦✐ss♦♥ ♣r♦❝❡ss ✇✐t❤ ❇r♦✇♥✐❛♥ ♠♦t✐♦♥

  ▲é✈② ❉r✐✈❡♥ ❖♣t✐♠❛❧ ❈♦♥tr♦❧

  L = (L t ) t≥ t t ) , P

  

✵ ✐s ❛♥ F ✲❛❞❛♣t❡❞ ▲é✈② ♣r♦❝❡ss ❞❡✜♥❡❞ ♦♥ Ω, F, (F t≥ ✳

✵

  ❚❤❡ st❛t❡ ♦❢ t❤❡ ❝♦♥tr♦❧❧❡❞ s②st❡♠✿ _{u u u u Z}

• dY = b (t, Y , u t ) dt + σ (t, Y , u t ) dB t γ (t, Y − , u , z) N (dt, dz) _{t t t t | {z } | {z } | {z } R n t} ⁻

  ❞r✐❢t ♣❛rt ❞✐✛✉s✐♦♥ ♣❛rt _{n n n n n×m} ❥✉♠♣ ♣❛rt ✇✐t❤ Y ✵ = y ∈ R ✱ b : [✵, ∞) × R × U → R ✱ σ : [✵, ∞) × R × U → R ✱ ❛♥❞ _{n n×m} γ : [

  ✵, ∞) × R × U → R ✳ ❈♦st ❢✉♥❝t✐♦♥❛❧✿ _τ

  Z _{u u} J [u] = E w (X , u ) ds + z (X ) , _{s s τ n} ✵ _u

  / ✇❤❡r❡ S ⊂ R ✱ w : S × U → R✱ z : ∂S → R✱ ❛♥❞ τ = ✐♥❢ {t > ✵ : X t ∈ S}✳

  ❚❤❛♥❦ ②♦✉ ❢♦r ②♦✉r ❛tt❡♥t✐♦♥✦