PRODUCT MOMENTS OF BIVARIATE RANDOM VARIABLES
Chapter 8 PRODUCT MOMENTS OF BIVARIATE RANDOM VARIABLES
In this chapter, we define various product moments of a bivariate random variable. The main concept we introduce in this chapter is the notion of covariance between two random variables. Using this notion, we study the statistical dependence of two random variables.
8.1. Covariance of Bivariate Random Variables First, we define the notion of product moment of two random variables
and then using this product moment, we give the definition of covariance between two random variables.
Definition 8.1. Let X and Y be any two random variables with joint density function f(x, y). The product moment of X and Y , denoted by E(XY ), is defined as
if X and Y are discrete
E(XY ) = x 2R X y 2R Y
: > R 1 R 1 xy f (x, y) dx dy if X and Y are continuous.
Here, R X and R Y represent the range spaces of X and Y respectively. Definition 8.2. Let X and Y be any two random variables with joint density
function f(x, y). The covariance between X and Y , denoted by Cov(X, Y ) (or XY ), is defined as
Cov(X, Y ) = E( (X µ X ) (Y µ Y ) ),
Probability and Mathematical Statistics
where µ X and µ Y are mean of X and Y , respectively. Notice that the covariance of X and Y is really the product moment of
X µ X and Y µ Y . Further, the mean of µ X is given by
Z 1 Z 1 Z 1
µ X = E(X) =
xf 1 (x) dx =
x f (x, y) dx dy,
and similarly the mean of Y is given by
Theorem 8.1. Let X and Y be any two random variables. Then
Cov(X, Y ) = E(XY ) E(X) E(Y ).
Proof:
Cov(X, Y ) = E((X µ X ) (Y µ Y ))
= E(XY µ X Y µ Y X+µ X µ Y )
= E(XY ) µ X E(Y ) µ Y E(X) + µ X µ Y
= E(XY ) µ X µ Y µ Y µ X +µ X µ Y
= E(XY ) µ X µ Y
= E(XY ) E(X) E(Y ).
Corollary 8.1. Cov(X, X) = 2 X .
Proof:
Cov(X, X) = E(XX) E(X) E(X) = E(X 2 )µ 2 X = V ar(X) = 2 X .
Example 8.1. Let X and Y be discrete random variables with joint density
( x+2y
What is the covariance XY between X and Y .
Product Moments of Bivariate Random Variables
Answer: The marginal of X is
Hence the expected value of X is
X 2
E(X) =
Similarly, the marginal of Y is
Hence the expected value of Y is
Further, the product moment of X and Y is given by
X 2 X 2
E(XY ) =
x y f (x, y)
x=1 y=1
= f(1, 1) + 2 f(1, 2) + 2 f(2, 1) + 4 f(2, 2)
Probability and Mathematical Statistics
Hence, the covariance between X and Y is given by
Cov(X, Y ) = E(XY ) E(X) E(Y )
Remark 8.1. For an arbitrary random variable, the product moment and covariance may or may not exist. Further, note that unlike variance, the covariance between two random variables may be negative.
Example 8.2. Let X and Y have the joint density function
What is the covariance between X and Y ? Answer: The marginal density of X is
Thus, the expected value of X is given by
Z 1
E(X) =
xf 1 (x) dx
Z 0
x (x + ) dx
0 2 x 3 x 2 1
Product Moments of Bivariate Random Variables
Similarly (or using the fact that the density is symmetric in x and y), we get
Now, we compute the product moment of X and Y .
Z 1 Z 1
E(XY ) =
(x 2 y+xy 2 ) dx dy
Hence the covariance between X and Y is given by
Cov(X, Y ) = E(XY ) E(X) E(Y )
Example 8.3. Let X and Y be continuous random variables with joint density function
What is the covariance between X and Y ? Answer: The marginal density of X is given by
Z 1x
f 1 (x) =
2 dy = 2 (1 x).
Probability and Mathematical Statistics
Hence the expected value of X is
Z 1 Z 1 1
µ X = E(X) =
Similarly, the marginal of Y is
Hence the expected value of Y is
The product moment of X and Y is given by
Z 1 Z 1x
E(XY ) =
Therefore, the covariance between X and Y is given by
Cov(X, Y ) = E(XY ) E(X) E(Y )
Product Moments of Bivariate Random Variables
Theorem 8.2. If X and Y are any two random variables and a, b, c, and d are real constants, then
Cov(a X + b, c Y + d) = a c Cov(X, Y ).
Proof:
Cov(a X + b, c Y + d)
= E ((aX + b)(cY + d)) E(aX + b) E(cY + d) = E (acXY + adX + bcY + bd) (aE(X) + b) (cE(Y ) + d) = ac E(XY ) + ad E(X) + bc E(Y ) + bd
[ac E(X) E(Y ) + ad E(X) + bc E(Y ) + bd] = ac [E(XY ) E(X) E(Y )] = ac Cov(X, Y ).
Example 8.4. If the product moment of X and Y is 3 and the mean of
X and Y are both equal to 2, then what is the covariance of the random
variables 2X + 10 and 5 2 Y+3?
Answer: Since E(XY ) = 3 and E(X) = 2 = E(Y ), the covariance of X and Y is given by
Cov(X, Y ) = E(XY ) E(X) E(Y ) = 3
Then the covariance of 2X + 10 and 5 2 Y + 3 is given by
2 Cov(X, Y ) 2 = ( 5) ( 1)
Remark 8.2. Notice that the Theorem 8.2 can be furthered improved. That is, if X, Y , Z are three random variables, then
Cov(X + Y, Z) = Cov(X, Z) + Cov(Y, Z)
and
Cov(X, Y + Z) = Cov(X, Y ) + Cov(X, Z).
Probability and Mathematical Statistics
The first formula can be established as follows. Consider
Cov(X + Y, Z) = E((X + Y )Z) E(X + Y ) E(Z)
= E(XZ + Y Z) E(X)E(Z) E(Y )E(Z) = E(XZ) E(X)E(Z) + E(Y Z) E(Y )E(Z) = Cov(X, Z) + Cov(Y, Z).
8.2. Independence of Random Variables In this section, we study the e↵ect of independence on the product mo-
ment (and hence on the covariance). We begin with a simple theorem. Theorem 8.3. If X and Y are independent random variables, then
E(XY ) = E(X) E(Y ).
Proof: Recall that X and Y are independent if and only if
f (x, y) = f 1 (x) f 2 (y).
Let us assume that X and Y are continuous. Therefore
Z 1 Z 1
E(XY ) =
= E(X) E(Y ).
If X and Y are discrete, then replace the integrals by appropriate sums to prove the same result.
Example 8.5. Let X and Y be two independent random variables with respective density functions:
g(y) =
0 otherwise .
Product Moments of Bivariate Random Variables
What is E X Y ?
Answer: Since X and Y are independent, the joint density of X and Y is given by
h(x, y) = f (x) g(y).
h(x, y) dx dy
f (x) g(y) dx dy
Remark 8.3. The independence of X and Y does not imply E E(X) X
= E(Y )
Y
but only implies E X = E(X) E Y 1 . Further, note that E Y Y 1 is not
equal to 1 E(Y ) . Theorem 8.4. If X and Y are independent random variables, then the
covariance between X and Y is always zero, that is
Cov(X, Y ) = 0.
Proof: Suppose X and Y are independent, then by Theorem 8.3, we have E(XY ) = E(X) E(Y ). Consider
Cov(X, Y ) = E(XY ) E(X) E(Y )
= E(X) E(Y ) E(X) E(Y ) = 0.
Example 8.6. Let the random variables X and Y have the joint density
What is the covariance of X and Y ? Are the random variables X and Y independent?
Probability and Mathematical Statistics
Answer: The joint density of X and Y are shown in the following table with
the marginals f 1 (x) and f 2 (y).
From this table, we see that
✓2 ◆ ◆ ✓2 1
0 = f(0, 0) 6= f 1 (0) f 2 (0) =
and thus
f (x, y) 6= f 1 (x) f 2 (y)
for all (x, y) is the range space of the joint variable (X, Y ). Therefore X and Y are not independent.
Next, we compute the covariance between X and Y . For this we need
Product Moments of Bivariate Random Variables
E(X), E(Y ) and E(XY ). The expected value of X is
X 1
E(X) =
xf 1 (x)
x= 1
= ( 1) f 1 ( 1) + (0)f 1 (0) + (1) f 1 (1)
Similarly, the expected value of Y is
= ( 1) f 2 ( 1) + (0)f 2 (0) + (1) f 2 (1)
The product moment of X and Y is given by
X 1 X 1
E(XY ) =
x y f (x, y)
x= 1 y= 1
= (1) f( 1, 1) + (0) f( 1, 0) + ( 1) f( 1, 1) + (0) f(0, 1) + (0) f(0, 0) + (0) f(0, 1) + ( 1) f(1, 1) + (0) f(1, 0) + (1) f(1, 1)
Hence, the covariance between X and Y is given by
Cov(X, Y ) = E(XY ) E(X) E(Y ) = 0.
Remark 8.4. This example shows that if the covariance of X and Y is zero that does not mean the random variables are independent. However, we know from Theorem 8.4 that if X and Y are independent, then the Cov(X, Y ) is always zero.
Probability and Mathematical Statistics
8.3. Variance of the Linear Combination of Random Variables Given two random variables, X and Y , we determine the variance of
their linear combination, that is aX + bY . Theorem 8.5. Let X and Y be any two random variables and let a and b
be any two real numbers. Then
V ar(aX + bY ) = a 2 V ar(X) + b 2 V ar(Y ) + 2 a b Cov(X, Y ).
Proof:
V ar(aX + bY )
⇣
2 ⌘
=E [aX + bY E(aX + bY )]
a E(X) 2 b E(Y )]
=a 2 E (X µ X ) 2 +b 2 E (X µ X ) 2 + 2 a b E((X µ X ) (Y µ Y ))
=a 2 V ar(X) + b 2 V ar(Y ) + 2 a b Cov(X, Y ).
Example 8.7. If V ar(X + Y ) = 3, V ar(X Y ) = 1, E(X) = 1 and E(Y ) = 2, then what is E(XY ) ?
Answer:
V ar(X + Y ) = 2 X + 2 Y + 2 Cov(X, Y ),
V ar(X Y)= 2 X + 2 Y
2 Cov(X, Y ).
Hence, we get
1 Cov(X, Y ) = [ V ar(X + Y ) V ar(X Y)]
Therefore, the product moment of X and Y is given by
E(XY ) = Cov(X, Y ) + E(X) E(Y )
Product Moments of Bivariate Random Variables
Example 8.8. Let X and Y be random variables with V ar(X) = 4,
V ar(Y ) = 9 and V ar(X Y ) = 16. What is Cov(X, Y ) ? Answer:
V ar(X Y ) = V ar(X) + V ar(Y )
2 Cov(X, Y )
16 = 4 + 9 2 Cov(X, Y ).
Hence
Cov(X, Y ) =
Remark 8.5. The Theorem 8.5 can be extended to three or more random variables. In case of three random variables X, Y, Z, we have
V ar(X + Y + Z) = V ar(X) + V ar(Y ) + V ar(Z)
+ 2Cov(X, Y ) + 2Cov(Y, Z) + 2Cov(Z, X).
To see this consider
V ar(X + Y + Z) = V ar((X + Y ) + Z) = V ar(X + Y ) + V ar(Z) + 2Cov(X + Y, Z) = V ar(X + Y ) + V ar(Z) + 2Cov(X, Z) + 2Cov(Y, Z) = V ar(X) + V ar(Y ) + 2Cov(X, Y )
+ V ar(Z) + 2Cov(X, Z) + 2Cov(Y, Z) = V ar(X) + V ar(Y ) + V ar(Z)
+ 2Cov(X, Y ) + 2Cov(Y, Z) + 2Cov(Z, X).
Theorem 8.6. If X and Y are independent random variables with E(X) =
0 = E(Y ), then
V ar(XY ) = V ar(X) V ar(Y ).
Proof:
V ar(XY ) = E (XY ) 2 2 (E(X) E(Y ))
= E (XY ) 2
=EX 2 Y 2
=EX 2 EY 2 (by independence of X and Y )
= V ar(X) V ar(Y ).
Probability and Mathematical Statistics
Example 8.9. Let X and Y be independent random variables, each with density
If the V ar(XY ) = 64 9 , then what is the value of ✓ ?
E(X) =
Since Y has the same density, we conclude that E(Y ) = 0. Hence
= V ar(XY )
= V ar(X) V ar(Y )
Hence, we obtain
4 ✓ p = 64 or ✓=2 2.
8.4. Correlation and Independence The functional dependency of the random variable Y on the random
variable X can be obtained by examining the correlation coefficient. The definition of the correlation coefficient ⇢ between X and Y is given below.
Definition 8.3. Let X and Y be two random variables with variances 2 X
and 2 Y , respectively. Let the covariance of X and Y be Cov(X, Y ). Then the correlation coefficient ⇢ between X and Y is given by
Cov(X, Y )
Theorem 8.7. If X and Y are independent, the correlation coefficient be- tween X and Y is zero.
Product Moments of Bivariate Random Variables
Proof:
Cov(X, Y ) ⇢=
Remark 8.4. The converse of this theorem is not true. If the correlation coefficient of X and Y is zero, then X and Y are said to be uncorrelated.
Lemma 8.1. If X ? and Y ? are the standardizations of the random variables
X and Y , respectively, the correlation coefficient between X ? and Y ? is equal to the correlation coefficient between X and Y .
Proof: Let ⇢ ?
be the correlation coefficient between X ? and Y ? . Further,
let ⇢ denote the correlation coefficient between X and Y . We will show that ⇢ ? = ⇢. Consider
This lemma states that the value of the correlation coefficient between two random variables does not change by standardization of them.
Theorem 8.8. For any random variables X and Y , the correlation coefficient ⇢ satisfies
1 ⇢ 1,
and ⇢ = 1 or ⇢ = 1 implies that the random variable Y = a X + b, where a and b are arbitrary real constants with a 6= 0.
Proof: Let µ X be the mean of X and µ Y
be the mean of Y , and 2 X and 2 Y
be the variances of X and Y , respectively. Further, let
⇤
X µ
Y = µ X and ⇤ = Y
X Y
Probability and Mathematical Statistics
be the standardization of X and Y , respectively. Then
V ar(X ⇤ Y ⇤ ) = V ar(X ⇤ ) + V ar(Y ⇤ ) 2Cov(X ⇤ ,Y ⇤ )
= 2 X ⇤ + 2 Y ⇤ 2⇢ ⇤ X ⇤ Y ⇤
= 1 + 1 2⇢ ⇤ = 1 + 1 2⇢
(by Lemma 8.1)
= 2(1 ⇢).
Since the variance of a random variable is always positive, we get
2 (1 ⇢) 0
which is
⇢ 1.
By a similar argument, using V ar(X ⇤ +Y ⇤ ), one can show that 1 ⇢. Hence, we have 1 ⇢ 1. Now, we show that if ⇢ = 1 or ⇢ = 1, then Y
and X are related through an affine transformation. Consider the case ⇢ = 1, then
V ar(X ⇤ Y ⇤ ) = 0.
But if the variance of a random variable is 0, then all the probability mass is concentrated at a point (that is, the distribution of the corresponding random variable is degenerate). Thus V ar(X ⇤ Y ⇤ ) = 0 implies X ⇤ Y ⇤ takes only one value. But E [X ⇤ Y ⇤ ] = 0. Thus, we get
Solving this for Y in terms of X, we get
Y=aX+b
Product Moments of Bivariate Random Variables
Thus if ⇢ = 1, then Y is a linear in X. Similarly, we can show for the case ⇢=
1, the random variables X and Y are linearly related. This completes the proof of the theorem.
8.5. Moment Generating Functions Similar to the moment generating function for the univariate case, one
can define the moment generating function for the bivariate case to com- pute the various product moments. The moment generating function for the bivariate case is defined as follows:
Definition 8.4. Let X and Y be two random variables with joint density
function f(x, y). A real valued function M : IR 2 ! IR defined by
M (s, t) = E e sX+tY
is called the joint moment generating function of X and Y if this expected value exists for all s is some interval h < s < h and for all t is some interval
k < t < k for some positive h and k.
It is easy to see from this definition that
From this we see that
for k = 1, 2, 3, 4, ...; and
2 M (s, t)
E(XY ) =
s t (0,0)
Example 8.10. Let the random variables X and Y have the joint density
( e y
for 0 < x < y < 1
f (x, y) =
0 otherwise.
Probability and Mathematical Statistics
What is the joint moment generating function for X and Y ? Answer: The joint moment generating function of X and Y is given by
M (s, t) = E e sX+tY
Z 1 Z 1
e sx+ty f (x, y) dy dx
e sx+ty e y dy dx
e sx+ty y dy dx
provided s + t < 1 and t < 1.
(1 s t) (1 t)
Example 8.11. If the joint moment generating function of the random variables X and Y is
M (s, t) = e (s+3t+2s 2 +18t 2 +12st)
what is the covariance of X and Y ? Answer:
Product Moments of Bivariate Random Variables
M (s, t) = e (s+3t+2s 2 +18t 2 +12st)
M = (1 + 4s + 12t) M(s, t)
= 1. M = (3 + 36t + 12s) M(s, t)
Now we compute the product moment of X and Y .
t s = (M(s, t) (1 + 4s + 12t))
t = (1 + 4s + 12t) M + M(s, t) (12).
s t (0,0)
Thus
E(XY ) = 15
and the covariance of X and Y is given by
Cov(X, Y ) = E(XY ) E(X) E(Y ) = 15 (3) (1) = 12.
Theorem 8.9. If X and Y are independent then
M aX+bY (t) = M X (at) M Y (bt),
Probability and Mathematical Statistics
where a and b real parameters. Proof: Let W = aX + bY . Hence
M aX+bY (t) = M W (t)
=Ee tW
⇣ t(aX+bY ) ⌘ =E e
=Ee taX e tbY
=Ee taX Ee tbY
(by Theorem 8.3)
=M X (at) M Y (bt).
This theorem is very powerful. It helps us to find the distribution of a linear combination of independent random variables. The following examples illustrate how one can use this theorem to determine distribution of a linear combination.
Example 8.12. Suppose the random variable X is normal with mean 2 and standard deviation 3 and the random variable Y is also normal with mean
0 and standard deviation 4. If X and Y are independent, then what is the probability distribution of the random variable X + Y ?
Answer: Since X ⇠ N(2, 9), the moment generating function of X is given by
M (t) = e µt+ 2 1 t X 2 =e 2t+ 9 2 t 2 .
Similarly, since Y ⇠ N(0, 16),
1 2 16 M 2
Y (t) = e µt+ 2 t =e 2 t .
Since X and Y are independent, the moment generating function of X + Y is given by
M X+Y (t) = M X (t) M Y (t) =e 2t+ 9 2 t 2 e 16 2 t 2 =e 2t+ 25 2 t 2 .
Hence X + Y ⇠ N(2, 25). Thus, X + Y has a normal distribution with mean
2 and variance 25. From this information we can find the probability density function of W = X + Y as
1 1 w 2 2
f (w) = p
e 2 ( 5 ) ,
1 < w < 1.
50⇡
Product Moments of Bivariate Random Variables
Remark 8.6. In fact if X and Y are independent normal random variables
with means µ X and µ Y and variances 2 X and 2 Y , respectively, then aX +bY is also normal with mean aµ X + bµ Y and variance a 22 X +b 22 Y .
Example 8.13. Let X and Y be two independent and identically distributed random variables. If their common distribution is chi-square with one degree of freedom, then what is the distribution of X + Y ? What is the moment generating function of X Y?
Answer: Since X and Y are both 2 (1), the moment generating functions
Since, the random variables X and Y are independent, the moment generat- ing function of X + Y is given by
Hence X + Y ⇠ 2 (2). Thus, if X and Y are independent chi-square random variables, then their sum is also a chi-square random variable.
Next, we show that X Y is not a chi-square random variable, even if
X and Y are both chi-square.
This moment generating function does not correspond to the moment gener- ating function of a chi-square random variable with any degree of freedoms. Further, it is surprising that this moment generating function does not cor- respond to that of any known distributions.
Remark 8.7. If X and Y are chi-square and independent random variables, then their linear combination is not necessarily a chi-square random variable.
Probability and Mathematical Statistics
Example 8.14. Let X and Y be two independent Bernoulli random variables with parameter p. What is the distribution of X + Y ?
Answer: Since X and Y are Bernoulli with parameter p, their moment generating functions are
M X (t) = (1 p) + pe t
M Y (t) = (1 p) + pe t .
Since, X and Y are independent, the moment generating function of their sum is the product of their moment generating functions, that is
Hence X + Y ⇠ BIN(2, p). Thus the sum of two independent Bernoulli random variable is a binomial random variable with parameter 2 and p.
8.6. Review Exercises
1. Suppose that X 1 and X 2 are random variables with zero mean and unit
variance. If the correlation coefficient of X 1 and X 2 is 0.5, then what is the
variance of Y = P 2 k 2
X k ?
k=1
2. If the joint density of the random variables X and Y is
what is the covariance of X and Y ? Are X and Y independent?
3. Suppose the random variables X and Y are independent and identically distributed. Let Z = aX + Y . If the correlation coefficient between X and
Z is 1 3 , then what is the value of the constant a ?
4. Let X and Y be two independent random variables with chi-square distri- bution with 2 degrees of freedom. What is the moment generating function of the random variable 2X + 3Y ? If possible, what is the distribution of 2X + 3Y ?
5. Let X and Y be two independent random variables. If X ⇠ BIN(n, p) and Y ⇠ BIN(m, p), then what is the distribution of X + Y ?
Product Moments of Bivariate Random Variables
6. Let X and Y be two independent random variables. If X and Y are both standard normal, then what is the distribution of the random variable
2 X 2 +Y 2 ?
7. If the joint probability density function of X and Y is
then what is the joint moment generating function of X and Y ?
8. Let the joint density function of X and Y be
What is the correlation coefficient of X and Y ?
9. Suppose that X and Y are random variables with joint moment generating function
for all real s and t. What is the covariance of X and Y ?
10. Suppose that X and Y are random variables with joint density function
What is the covariance of X and Y ? Are X and Y independent?
11. Let X and Y be two random variables. Suppose E(X) = 1, E(Y ) = 2,
V ar(X) = 1, V ar(Y ) = 2, and Cov(X, Y ) = 1 2 . For what values of the
constants a and b, the random variable aX + bY , whose expected value is 3, has minimum variance?
12. A box contains 5 white balls and 3 black balls. Draw 2 balls without replacement. If X represents the number of white balls and Y represents the number of black balls drawn, what is the covariance of X and Y ?
13. If X represents the number of 1’s and Y represents the number of 5’s in three tosses of a fair six-sided die, what is the correlation between X and Y ?
Probability and Mathematical Statistics
14. Let Y and Z be two random variables. If V ar(Y ) = 4, V ar(Z) = 16, and Cov(Y, Z) = 2, then what is V ar(3Z 2Y )?
15. Three random variables X 1 ,X 2 ,X 3 , have equal variances 2 and coef- ficient of correlation between X 1 and X 2 of ⇢ and between X 1 and X 3 and
between X 2 and X 3 of zero. What is the correlation between Y and Z where
Y=X 1 +X 2 and Z = X 2 +X 3 ?
16. If X and Y are two independent Bernoulli random variables with pa- rameter p, then what is the joint moment generating function of X Y?
17. If X 1 ,X 2 , ..., X n are normal random variables with variance 2 and covariance between any pair of random variables ⇢ 2 , what is the variance
of 1 n (X 1 +X 2 +···+X n )?
18. The coefficient of correlation between X and Y is 1 3 and 2 X = a,
Y = 4a, and 2 Z = 114 where Z = 3X 4Y . What is the value of the
constant a ?
19. Let X and Y be independent random variables with E(X) = 1, E(Y ) =
2, and V ar(X) = V ar(Y ) = 2 . For what value of the constant k is the
expected value of the random variable k(X 2 Y 2 )+Y 2 equals 2 ?
20. Let X be a random variable with finite variance. If Y = 15
X, then
what is the coefficient of correlation between the random variables X and (X + Y )X ?
Conditional Expectations of Bivariate Random Variables