Introduction to Probability

Pea taxi will take more than 5 minutes to arrive) = �.

AI's waiting time, call i t X, is a mixed random variable. With probability

5 peA) = - + 3 3 -.-=- 2 6'

it is equal to its discrete component Y (corresponding to either finding a taxi waiting, or boarding the bus), which has PMF

{ if

if Y = 0,

py (y)

Y = 5,

if y= 0,

if y= 5.

15 '

[This equation follows from the calculation

0, A)

py (0) = P(Y = 0I A) = P(Y =

peA) - 3P(A) ' The calculation for py (5) is similar.] With the complementary probability 1 - peA),

the waiting time is equal to its continuous component Z (corresponding to boarding a taxi after having to wait for some time less than 5 minutes), which has PDF

1/5, if 0 :::; � :::; 5,

fz(z) = {

0, otherwIse.

The CDF is given by Fx (x) = P(A)Fy (x) + (1 - P(A) ) Fz (x), from which

{ 5 12 1 x

if x < 0,

Fx (x) = _ . _ + _ . - if O :::;x < 5,

6 15 6 5'

if 5 :::; x.

The expected value of the waiting time is

E[X] = P(A)E[Y] + ( 1 - P(A)) E[Z] = - . -.5+ - . - = -.

188 General Random Variables Chap. 3

Problem 10. * Simulating a continuous random variable.

A computer has a

subroutine that can generate values of a random variable U that is uniformly distributed in the interval [0, 1] . Such a subroutine can be used to generate values of a continuous random variable with given CDF F(x) as follows. If U takes a value u, we let the value

of X be a number x that satisfies F(x) = u. For simplicity, we assume that the given CDF is strictly increasing over the range S of values of interest. where S = {x I 0 < F( x) < I}. This condition guarantees that for any u E (0. 1). there is a unique x that

satisfies F(x) = u.

( a ) Show that the CDF of the random variable X thus generated is indeed equal to

the given CDF. ( b ) Describe how this procedure can be used to simulate an exponential random

variable with parameter 'x. ( c ) How can this procedure be generalized to simulate a discrete integer-valued ran­

dom variable?

Solution. ( a ) By definition, the random variables X and U satisfy the relation F(X) =

U. Since F is strictly increasing, we have for every x,

X�x if and only if

F(X) � F(x).

Therefore,

P(X � x) = P ( F(X) � F(x) ) = p (U � F(x) ) = F(x) ,

where the last equality follows because U is uniform. Thus, X has the desired CDF.

( b ) The exponential CDF has the form F(x) = 1 - e->'x for x � O. Thus, to generate values of X. we should generate values u E (0. 1) of a uniformly distributed random

variable U. and set X to the value for which 1 - e- >'x u, or = x n u ,x. = - I (1 - ) /

( c ) Let again F be the desired CDF. To any u E (0. 1), there corresponds a unique

integer Xu such that F(xu - 1 ) < u � F(xu). This correspondence defines a random variable X as a function of the random variable U. We then have. for every integer k.

P(X = k) = P ( F(k - 1) < U � F(k) ) = F(k) - F(k - 1).

Therefore, the CDF of X is equal to F, as desired.

SECTION 3.3. Normal Random Variables Problem 11. Let X and Y be normal random variables with means 0 and 1, respec­

tively, and variances 1 and 4, respectively.

( a ) Find P(X � 1 .5) and P(X � -1). ( b ) Find the PDF of ( Y - 1 )/2. ( c ) Find P(-1 :$ Y :$ 1).

Problem 12. Let X be a normal random variable with zero mean and standard deviation u. Use the normal tables to compute the probabilities of the events {X � ku} and { I XI � ku} for k = 1 , 2, 3.

Problems 189

Problem 13. A city's temperature is modeled as a normal random variable with mean and standard deviation both equal to 10 degrees Celsius. What is the probability that

the temperature at a randomly chosen time will be less than or equal to 59 degrees Fahrenheit?

Problem 14. * Show that the normal PDF satisfies the normalization property. Hint:

The integral J�x e-x /2 dx is equal to the square root of

00 2 1= 1 e -x /2 e

-y 2 /2 d d x y,

-x - 00

and the latter integral can b e evaluated by transforming to polar coordinates. Solution. We note that

21T"

1 1= e -r /2r dr dO

= - 1 271"

where for the third equality, we use a transformation into polar coordinates, and for the fifth equality, we use the change of variables u = r2 / . Thus, we have

1= e -x2/2 d - 1 - x-

..j'i;

because the integral is positive. Using the change of variables u (x - = J-l)/u, it follows that

= lOG

I IX = _ =

fx (x) dx

e-u /2du = 2 = 1.

e-(x-/L) /217 dx

v'2ii u

..j'i;

- :x;

SECTION 3.4. Joint PDFs of Multiple Random Variables Problem 15. A point is chosen at random (according to a uniform PDF) within a

semicircle of the form { (x, y) I x2 + y2 :::; r2, y � O}, for some given r > O.

(a) Find the joint PDF of the coordinates X and Y of the chosen point.

190 General Random Variables Chap. 3

( b ) Find the marginal PDF of Y and use it to find E[Y] . ( c ) Check your answer in ( b ) by computing E[Y] directly without using the marginal

PDF of Y. Problem 16. Consider the following variant of Buffon's needle problem ( Example

3. 1 1 ) ' which was investigated by Laplace. A needle of length I is dropped on a plane surface that is partitioned in rectangles by horizontal lines that are a apart and vertical

lines that are b apart. Suppose that the needle's length I satisfies 1 < a and 1 < b. What

is the expected number of rectangle sides crossed by the needle? What is the probability that the needle will cross at least one side of some rectangle?

Problem 17.* Estimating an expected value by simulation using samples of another random variable. Let Y1 •••• , Yn be independent random variables drawn from a common and known PDF fy. Let S be the set of all possible values of Yi,

S = {y I fy(y) > O}. Let X be a random variable with known PDF fx, such that fx(y) = 0, for all y ¢ S. Consider the random variable

Show that

E[Z] = E[X] .

Solution. We have

Yi fY(Yi) = E[X] .

[ fY ( Y i ) ] n i=1

Thus,

E[Z] = .!. E Yi fx(Yi) = .!. E[X] = E[X] .

n t=1

SECTION 3.5. Conditioning

Problem 18. Let X be a random variable with PDF

otherwise, x

if 1 < ::; fx (x) 3, { X/4,

and let A be the event {X � 2}. ( a ) Find E[X] ,

and E[X P(A), fXIA(X), I A] . ( b ) Let Y = X . Find E[Y] and var ( Y ) .

Problem 19. The random variable X has the PDF

x {cx-2, 0, otherwIse.

if 1 ::; � ::; 2,

fx( ) =

Problems 191 (a) Determine the value of c.

(b) Let A be the event {X > 1.5}. Calculate P(A) and the conditional PDF of X

given that A has occurred.

(c) Let Y = X2 . Calculate the conditional expectation and the conditional variance

of Y given A.

Problem 20. An absent-minded professor schedules two student appointments for the same time. The appointment durations are independent and exponentially distributed with mean thirty minutes. The first student arrives on time, but the second student arrives five minutes late. What is the expected value of the time between the arrival of the first student and the departure of the second student?

Problem 2 1 . We start with a stick of length e. We break it at a point which is chosen according to a uniform distribution and keep the piece, of length Y, that contains the

left end of the stick. We then repeat the same process on the piece that we were left with, and let X be the length of the remaining piece after breaking for the second time.

(a) Find the joint PDF of Y and X. (b) Find the marginal PDF of X.

(c) Use the PDF of X to evaluate E[X]. (d) Evaluate E[X] , by exploiting the relation X = y. (X/ Y) .

Problem 22. We have a stick o f unit length, and we consider breaking i t i n three pieces using one of the following three methods.

(i) We choose randomly and independently two points on the stick using a uniform PDF, and we break the stick at these two points.

(ii) We break the stick at a random point chosen by using a uniform PDF, and then

we break the piece that contains the right end of the stick. at a random point chosen by using a uniform PDF.

(iii) We break the stick at a random point chosen by using a uniform PDF, and then we break the larger of the two pieces at a random point chosen by using a uniform PDF.

For each of the methods (i) , (ii), and (iii) , what is the probability that the three pieces we are left with can form a triangle?

Problem 23. Let the random variables X and Y have a joint PDF which is uniform over the triangle with vertices at (0, 0) , (0, 1 ) , and ( 1 . 0).

(a) Find the joint PDF of X and Y. (b) Find the marginal PDF of Y .

(c) Find the conditional PDF of X given Y . (d) Find E [X I Y = yj , and use the total expectation theorem t o find E [X] i n terms

of E [Y] . (e) Use the symmetry of the problem to find the value of E[X].

192 General Random Variables Chap. 3

Problem 24. Let X and Y be two random variables that are uniformly distributed over the triangle formed by the points (0, 0), (1. 0) , and (0, 2) (this is an asymmetric version of the PDF in the previous problem) . Calculate E[X] and E[Y] by following the same steps as in the previous problem.

Problem 25. The coordinates X and Y of a point are independent zero mean normal random variables with common variance (72. Given that the point is at a distance of at least c from the origin. find the conditional joint PDF of X and Y.

Problem 26.* Let

XI, . . . . Xn be independent random variables. Show that

( TIn X ) n ,=1 ' var ( ( )

X, + )

var

TI�=1 E[X;] g E[Xll

1 - 1.

Solution. We have var

(IT X,)

[IT X;] - IT (E[XiJ)2

i=1

i= 1 - ,=1

i=1

11 n

= II E [X;] II (E[X,J) 2

- i=1

II var(X, ) ( + (E[X,J) 2) II (E[XiJ) 2.

The desired result follows by dividing both sides by

Problem 21.* Conditioning multiple random variables on events. Let X and Y be continuous random variables with joint PDF fx.y , let A be a subset of the two-dimensional plane, and let C = {(X, Y) E A}. Assume that P(C) > 0, and define

fx.Ylc(x, y) = {

fx.Y (x, y) if

(x,

y) E A,

P(C) ,

otherwise.

(a) Show that fX.Ylc is a legitimate joint PDF. (b) Consider a partition of the two-dimensional plane into disjoint subsets Ai , i =

1, . . . , n, let Ci = {(X. Y) E Ad, and assume that P(Ci) > 0 for all i. Derive

the following version of the total probability theorem

fx,y(x, y) = L P(C)fX.YICi (x, y) .

i=1

Problems 193

{ f p)..e-AX, X

Problem 28.* Consider the following two-sided exponential PDF

if x � 0,

(x) = (1 _ p) .. e AX, if x < 0,

where ).. and p are scalars with )" > ° and p E [0, 1]. Find the mean and the variance of X in two ways:

(a) By straightforward calculation of the associated expected values. (b) By using a divide-and-conquer strategy, and the mean and variance of the (one­

sided) exponential random variable. Solution. (a)

f:

E[X] = xfx(x) dx

1° x(1 - p) .. eAX dx - 100 xp)..e-AX dx

p - 1 )..

f:

E[X2] = x2 fx (x) dx

= 1° x2(1 - p) .. eAX dx + 100 x2p)..e-AX dx

2 ( 1 - p) 2 p

2 )..2 '

and

2 (2P - 1

var(X) =

(b) Let A be the event {X � O}, and note that P(A) = p. Conditioned on A, the

random variable X has a (one-sided) exponential distribution with parameter )... Also, conditioned on AC• the random variable -X has the same one-sided exponential dis­ tribution. Thus,

E[X I A] = >:'

and It follows that

E[X] = P(A)E[X I A] + P(AC)E[X l AC]

p 1-p -

).. - --

p - l )..

194 General Random Variables Chap. 3

E[X2] = P(A)E[X2 1 AI + P(AC)E[X2 1 AC]

2p

2( 1 - p)

2 - .;\2 '

and

2 (2P -

var(X)

Problem 29.* Let X, Y. and Z be three random variables with joint PDF /x.y.z . Show the multiplication rule:

/x.Y.z (x, Y, z) = /x l Y,z (x I y, z)/Yl z (Y I z)/z ( z) .

Solution. We have, using the definition of conditional density,

x,Y, z (x, y, z

/ x I y,Z x Y, z - ( I ) -/

/ Y.Z Y, Z ( ) ,

and

/Y,z (y, z) = /Y IZ (Y I z)/z (z) .

Combining these two relations, we obtain the multiplication rule.

/x (x) = { 1 B(o:, ,8)

Problem 30.* The Beta PDF. The beta PDF with parameters Q > 0 and {3 > 0

has the form

xo - 1 ( 1 _ X)i3- 1

if 0 :5 x :5 1,

otherwise.

The normalizing constant is

and is known as the Beta function.

(a) Show that for any m > 0, the mth moment of X is given by

B(o: + m, (3)

E[X7n] =

B(o:, ,8) .

(b) Assume that 0: and (3 are integer. Show that

(0: - 1)' ((3 I ) ! B(o:, (3) . = (0: + (3 - 1 ) !

so that

. (0: + m - 1 )

E[X 1 =

( 0: + (3)(0: + (3 + 1 ) . . . (0: + (3 + m - 1)

Problems 195

(Recall here the convention that O! = 1.)

Solution. ( a) We have

11 x )13- 1 d X =

E[Xm]

x x m 0- 1 (1 _

B(o + )3)

m,

B(o, )3) 0 B(o. )3) ·

(b) In the special case where 0 = 1 or )3 = 1, we can carry out the straightforward

integration in the definition of B(o, )3), and verify the result. We will now deal with the general case. Let Y, Yl , . . . , Yo+13 be independent random variables, uniformly dis­

tributed over the interval [0, 1], and let A be the event

Then,

P(A)

(0 + )3 + 1)! '

because all ways of ordering these 0 + )3 + 1 random variables are equally likely.

Consider the following two events:

B = { max { Y1 , . . . , Yo } �Y } ,

We have, using the total probability theorem,

p(B n C) =

p(B n C I Y = y)Jy (y) dy

1 = p( max { Y1 , . . . , Yo }�y� min { Yo+1 , . . . , Yo+13 } ) dy

= p( max { E . . . . , Yo } :-; y) ply :-; min { Yo+J , . . . ' yo+, } ) dy

/.'

= yO (1 y)" dy.

We also have

p(A I B n C) =

0! 1'! '

because given the events B and C, all o! possible orderings of Y1 , ••• , Yo are equally likely, and all f'! possible orderings of Yo+! , . . . , Yo +13 are equally likely.

By writing the equation

P(A) = P(B n C) P(A I B n C)

in terms of the preceding relations, we finally obtain

/.'

",1m

yO (1 - y)" dy,

+ + I)' =

196 General Random Variables Chap. 3

11 Y (1 - y) dy = o .

or

f3 0:. . ' {3'

(0: + (3 + I)!

This equation can be written as

B(o: + 1. {3 + 1) =

0: > 0, ( (3 > O.

0: + + 1 . (3 " for all integer )

o:! B!

Problem 31.* Estimating an expected value by simulation. Let Ix (x) be a

PDF such that for some nonnegative scalars a, b, and c, we have Ix (x) = 0 for all x outside the interval [a, b]. and xix (x) � c for all x. Let Yr , i = 1 , . . . , n, be independent

random variables with values generated as follows: a point (Vi, Wt) is chosen at random (according to a uniform PDF) within the rectangle whose corners are (a, 0) , (b, 0) . (a, c ) , and (b, c ) , and if W1 � Vi l x ( Vi ) . the value of Yi is set to 1, and otherwise it is set to O. Consider the random variable

Y1 +...+

Show that

E [ X E[Z] ] =

c (b - a)

and

var(Z) 1

4n '

In particular, we have var(Z) 0 as n 00.

--+

--+

Solution. We have

P(Yi = 1) = P (Wt � Vi Ix (Vi))

-lb 1V1X(V) a 0 c (

1 dw dv

b - a)

vtx (v) dv

c(b - a)

E [ X - c(b - a)' ]

The random variable Z has mean P ( Yi = 1) and variance

P ( Yr = 1) (1 - P ( Yi = 1))

var(Z) =

Since 0 :::; (1 - 2p)2 = 1 - 4p( 1 - p), we have p(1 - p) � 1/4 for any p in [0, 1]' so it follows that var(Z) :::; 1/(4n).

Problem 32.* Let X and Y be continuous random variables with joint PDF Ix ,Y . Suppose that for any subsets A and B of the real line, the events { X E A} and {Y E B}

are independent. Show that the random variables X and Y are independent.

Problems 197

Solution. For any two real numbers x and y, using the independence of the events {X � x} and {Y � y}, we have

Fx.y (x, y) = P ( X� x, Y� y) = P ( X� x) P(Y � y) = Fx (x)Fy (y).

Taking derivatives of both sides, we obtain

82 Fx. y

8Fx 8Fy

= fx (x)fy (y),

fx.y (x, y)

8x8y (x, y) 8x (x) 8y (y)

which establishes that X and Y are independent. Problem 33. * The sum of a random number of random variables. You visit

a random number N of stores and in the ith store, you spend a random amount of money Xi. Let

T = Xl + +... + X2 XN

be the total amount of money that you spend. We assume that N is a positive integer random variable with a given PMF, and that the Xl are random variables with the same mean E[X] and variance var(X). Furthermore, we assume that N and all the Xi are independent. Show that

E[T] = E[X] E[N], and var(T) = var(X) E[N] + ( E[X] ) 2 var(N). Solution. We have for all i,

E[T I N = i] = iE[X],

since conditional on N = i, you will visit exactly i stores, and you will spend an expected amount of money E[X] in each.

We now apply the total expectation theorem. We have

E[T] L P ( N = i) E[T I N = i]

i=l oc

= L P ( N = i)iE[X]

i=l

= E[X] L i P ( N = i)

= E[X] E[N].

198 General Random Variables Chap. 3

Similarly, using also the independence of the Xi, which implies that E[XiXj] = (E[XJ)2

if i =I- j, the second moment of T is calculated as

00 E[ T 2 ] = L P ( N =i ) E[ T 2 1 N = i]

= L P ( N = i) ( i E[ X 2 ] + i(i - 1) (E [X J)2)

t=1 00

= E[X2] L

iP(N = i) + ( E[XJ ) 2L i(i - ) P ( N i) =

i=1

t=1

E[X2] E[N] ( E [X J) 2 (E[N2] - E[NJ)

= var(X) E[N] + (E[XJ)2E[N2].

The variance is then obtained by

var(T) = E[T2] (E[TJ) 2 = var(X) E[N] + (E[XJ)2E[N2] - (E[XJ) 2 (E[Nl) 2 = var(X) E[N] + (E[XJ)\E[N2] - (E[NJ)2),

so finally

var(T) = var(X) E[N] + (E[Xl) 2 var(N).

Note: The formulas for E[T] and var(T) will also be obtained in Chapter 4, using a more abstract approach.

SECTION 3.6. The Continuous Bayes' Rule

Problem 34. A defective coin minting machine produces coins whose probability of heads is a random variable P with PDF

jp(p) = peP, p E { [O, �],

0, otherwise.

A coin produced by this machine is selected and tossed repeatedly, with successive tosses assumed independent.

(a) Find the probability that a coin toss results in heads. (b) Given that a coin toss resulted in heads, find the conditional PDF of P.

(c) Given that the first coin toss resulted in heads, find the conditional probability

of heads on the next toss. Problem 35. * Let X and Y be independent continuous random variables with PDFs

j x and jy, respectively, and let Z = X + Y.

Problems 199

(a) Show that fZlx (z I x) = fy (z - x). Hint: Write an expression for the conditional CDF of Z given X, and differentiate.

(b) Assume that X and Y are exponentially distributed with parameter A. Find the conditional PDF of X, given that Z = z.

(c) Assume that X and Y are normal random variables with mean zero and variances

(7; and (7�, respectively. Find the conditional PDF of X, given that Z = z.

Solution. (a) We have

P(Z :::; z IX = x) = P(X + Y :::; z I X =x ) = P(x + Y :::; z I X = x) = P(x + Y :::; z)

= P(Y :::; z - x),

where the third equality follows from the independence of X and Y. By differentiating both sides with respect to z, the result follows.

( b ) We have, for 0 :::; x :::; z,

fy (z - x) f x (x) Ae->'(z-x) Ae- >'x XIZx z - f (I) - fZlx (z I x)fx (x)

A2e- >'z

fz (z)

fz (z)

fz (z)

- fz (z) .

Since this is the same for all x, it follows that the conditional distribution of X is

uniform on the interval [0, z] , with PDF fXlz(x I z) = liz.

(c ) We have

fy (z - x)fx (x)

f X IZ x z - (I) -

fz (z)

We focus on the terms in the exponent. By completing the square, we find that the negative of the exponent is of the form

( Z(7; x- ) + Z2 - ( I -

- + x2 (7; + (7�

2(7; - 2(7;(7� - -

(7; + (7�

2(7�

Thus, the conditional PDF of X is of the form

. (7 x (7 y exp { - ( X Z(7 x )}

f x I z (x I z ) = c( z)

(7x(7y 2 2 2

(7x (7y

where c(z) does not depend on x and plays the role of a normalizing constant. We recognize this as a normal distribution with mean

E (72 [ X

I Z = z] =

2 + (7y (7x 2 Z,

and variance

var(X (7;(7�

IZ = z) =

(7x 2 + (7y 2'