Therefore, E[Yl l Y] = Yin. (b) We can think of a and W as sums of independent standard normal random vari
Therefore, E[Yl l Y] = Yin. (b) We can think of a and W as sums of independent standard normal random vari
ables:
W = Wl + " ' + Wm,
We identify Y with a + W and use the result from part (a), to obtain
8+W
E[ 8d e + W] =
k +m
Thus,
E[e I e k + W]
= E[e} +... + ek I e + W] =
(8 + W).
k +m
Sec. 8.5 Problems
449 The formula for the conditional mean derived in Example 8.3, specialized to the
current context (zero prior mean and a single measurement) shows that the conditional expectation is of the form
consistent with the answer obtained here. (c) We recall that the sum of independent Poisson random variables is Poisson. Thus
the argument in part (b) goes through, by thinking of 8 and W as sums of A (respec tively, J.t) independent Poisson random variables with mean one. We then obtain
E[8 1 8 + W] =
SECTION 8.4. Bayesian Linear Least Mean Squares Estimation
Problem 14. Consider the random variables e and X in Example 8. 11. Find the linear LMS estimator of 8 based on X, and the associated mean squared error.
Problem 15. For the model in the shopping cart problem (Problem 11), derive and plot the conditional mean squared error, as a function of the number on the observed
cart, for the MAP, LMS, and linear LMS estimators.
/x,e (x, O) = {OC" otherwise,
Problem 16. The joint PDF of random variables X and e is of the form
if (x, O) E S,
where c is a constant and S is the set
S = { (x, O) I 0 � x � 2, 0 � 0 � 2, x-I � 0 � x}.
We want to estimate e based on X.
(a) Find the LMS estimator g(X) of e. (b) Calculate E [(e - g(X»2 1 X = x] , E [g(X)] , and var (g(X» . (c) Calculate the mean squared error E [(e-g(X» 2] . Is it the same as E [ var(e I X)] ?
(d) Calculate var(8) using the law of total variance.
(e) Derive the linear LMS estimator of e based on X, and calculate its mean squared
error.
Problem 17. Let e be a positive random variable, with known mean J..L and variance (12, to be estimated on the basis of a measurement X of the form X = v'8 W. We assume that W is independent of e with zero mean, unit variance, and known fourth
moment E[W4]. Thus, the conditional mean and variance of X given e are 0 and
450 Bayesian Statistical Inference Chap. 8
8, respectively, so we are essentially trying to estimate the variance of X given an observed value. Find the linear LMS estimator of e based on X, and the linear LMS
estimator of 8 based on X2.
A doctor is treating a patient who has accidentally swallowed a needle. The key factor in whether to operate on the patient is the length e of the needle, which is unknown, but is assumed to be uniformly
Problem 18. Swallowed Buft'on's needle.
distributed between 0 and I > O. We wish to form an estimate of 8 based on X, its
projected length in an X-ray. We introduce a two-dimensional coordinate system and write
X = e cos w,
where W is the acute angle formed by the needle and one of the axes. We assume that W is uniformly distributed in the interval [0, 7r /2] ' and is independent from e.
(a) Find the LMS estimator E [ e I Xl. In particular, derive FX le(x I 0) , fX le(x I 0) , fx ( x ) , felx (O I x ) , and then compute E[e IX = x l . Hint: You may find the
following integration formulas useful:
a. do. I =
a I a - c2 0.2 - c21 . a
a (a.
c2) I
- c2
do. = log +
(b) Find the linear LMS estimate of e based on X, and the associated mean squared
error.
Problem 19. Consider a photodetector in an optical communications system that
counts the number of photons arriving during a certain interval. A user conveys infor mation by switching a photon transmitter on or off. Assume that the probability of
the transmitter being on is p. If the transmitter is on, the number of photons trans mitted over the interval of interest is a Poisson random variable e with mean ).. If the transmitter is off, the number of photons transmitted is zero.
Unfortunately, regardless of whether or not the transmitter is on or off, photons may still be detected due to a phenomenon called "shot noise." The number N of detected shot noise photons is a Poisson random variable with mean /1. Thus, the total
number X of detected photons is equal to e + N if the transmitter is on, and is equal
to N otherwise. We assume that N and 8 are independent, so that e + N is also Poisson with mean ). + /1.
(a) What is the probability that the transmitter was on, given that the photodetector
detected k photons? (c) Describe the MAP rule for deciding whether the transmitter was on. (d) Find the linear LMS estimator of the number of transmitted photons, based on
the number of detected photons.
Problem 20.* Estimation with spherically invariant PDFs. Let 8 and X be
continuous random variables with joint PDF of the form
fe.x (0, x) = h( q(O, x)) ,
where h is a nonnegative scalar function, and q(O, x) is a quadratic function of the form
q(O, x) = a(O - 0)2 + b(x - X)2 - 2 c ( 0 - O)(x - x).
Sec. 8.5 Problems
451 Here a, b, c, 8, x are some scalars with a j O. Derive the LMS and linear LMS estimates,
for any x such that E[e I X = x] is well-defined and finite. Assuming that q(O, x) �0 for all x, 0, and that h is monotonically decreasing, derive the MAP estimate and show
that it coincides with the LMS and linear LMS estimates. Solution. The posterior is given by
= h ( q(O, x) )
e , x (O , x)
f x (x) x (x) ·
(0 1 ) x
To motivate the derivation of the LMS and linear LMS estimates, consider first the
MAP estimate, assuming that q(O, x) � 0 for all x, 0, and that h is monotonically
decreasing. The MAP estimate maximizes h( q(O, x) ) and, since h is a decreasing
function, it minimizes q( 0, x) over O. By setting to 0 the derivative of q( 0, x) with respect to 0, we obtain
o = 0+ - (x - x).
(We are using here the fact that a nonnegative quadratic function of one variable is
minimized at a point where its derivative is equal to 0.)
We will now show that
B is equal to the LMS and linear LMS estimates [without
the assumption that q(O, x) �0 for all x, 0, and that h is monotonically decreasing].
We write
0-0 c
= 0-0+ - (x - x),
and substitute in the formula for q(O, x) to obtain after some algebra
( b - -;;: (x ) - x ) .
2 C q(O, x) = a(O - O) 2 +
Thus, for any given x, the posterior is a function of 0 that is symmetric around 8. This
implies that
B is equal to the conditional mean E[e I X = x] , whenever E[e I X = x] is well-defined and finite. Furthermore, we have
E[e I X] c
= 0+ - (X - x).
Since E[e I X] is linear in X, it is also the linear LMS estimator. Problem 21. * Linear LMS estimation based on two observations. Consider
three random variables e, X, and Y, with known variances and covariances. Assume
that var(X) > 0, var(Y) > 0, and that I p(X, Y) I =j:. 1. Give a formula for the linear
LMS estimator of e based on X and Y, assuming that X and Y are uncorrelated, and also in the general case.
Solu tion. We consider a linear estimator of the form e = aX + bY + c and choose a,
b, and c to minimize the mean squared error E
[ 2 (e - aX - bY - C ) ] . Suppose that a
[ 2 (8 - aX - bY - C ) ] , so
and b have already been chosen. Then, c must minimize E
c = E[e] - aE[X] - bErYl.
452 Bayesian Statistical Inference Chap. 8
It follows that a and b minimize
E [( (8 - E[8]) - a(X - E[X]) - b(Y - E[Y])) 2 ] .
We may thus assume that 8, X, and Y are zero mean, and in the final formula subtract the means. Under this assumption, the mean squared error is equal to
E [(8 - aX - by)2] = E[82] + a2E[X2] + b2E[y2] _ 2aE[8X] - 2bE[8Y] + 2abE[XY]. Assume that X and Y are uncorrelated, so that E[XY] = E[X] E[Y] = O. We
differentiate the expression for the mean squared error with respect to a and b, and set the derivatives to zero to obtain
_ E[8X] _ cov(8, X) cov(8, Y) a-
b = E[8Y]
E[Y2] var(Y) . Thus, the linear LMS estimator is
E[X2] var(X)
8 = E[8] + cov(8, X) (X _ E[Xl) + cov(8, Y) (Y - E[Y]).
var(X)
var(Y)
If X and Y are correlated, we similarly set the derivatives of the mean squared er ror to zero. We obtain and then solve a system of two linear equations in the unknowns
a and b, whose solution is
a --
b = var(X)cov(8, Y) - cov(8, X)cov(X, Y) . var(X)var(Y) - cov2(X, Y)
Note that the assumption Ip(X, Y) I =/:. 1 guarantees that the denominator in the pre ceding two equations is nonzero.