Point Estimation and Interval Estimation

3.1 Point Estimation and Interval Estimation

Imagine that someone wanted to weigh a certain object using spring scales. The object has an unknown weight, ω. The weight measurement, performed with the scales, has usually two sources of error: a calibration error, because of the spring’s

82 3 Estimating Data Parameters

loss of elasticity since the last calibration made at the factory, and exhibiting, therefore, a permanent deviation (bias) from the correct value; a random parallax error, corresponding to the evaluation of the gauge needle position, which can be considered normally distributed around the correct position (variance). The situation is depicted in Figure 3.1.

The weight measurement can be considered as a “bias + variance” situation. The bias, or systematic error, is a constant. The source of variance is a random error.

bias

Figure 3.1. Measurement of an unknown quantity ω with a systematic error (bias) and a random error (variance σ 2 ). One measurement instance is w.

Figure 3.1 also shows one weight measurement instance, w. Imagine that we performed a large number of weight measurements and came out with the average value of w . Then, the difference ω − w measures the bias or accuracy of the weighing device. On the other hand, the standard deviation, σ, measures the precision of the weighing device. Accurate scales will, on average, yield a measured weight that is in close agreement with the true weight. High precision scales yield weight measurements with very small random errors.

Let us now turn to the problem of estimating a data parameter, i.e., a quantity θ characterising the distribution function of the random variable X, describing the data. For that purpose, we assume that there is available a random sample x = [

x 1 , x 2 , K , x n ] ’ − our dataset in vector format −, and determine a value t n (x), using

an appropriate function t n . This single value is a point estimate of θ.

The estimate t n (x) is a value of a random variable, that we denote T, called point estimator or statistic, T ≡ t n (X), where X denotes the n-dimensional random variable corresponding to the sampling process. The point estimator T is, therefore,

a random variable function of X. Thus, t n (X) constitutes a sort of measurement device of θ. As with any measurement device, we want it to be simultaneously accurate and precise. In Appendix C, we introduce the topic of obtaining unbiased and consistent estimators. The unbiased property corresponds to the accuracy notion. The consistency corresponds to a growing precision for increasing sample sizes.

3.1 Point Estimation and Interval Estimation

When estimating a data parameter the point estimate is usually insufficient. In fact, in all the cases that the point estimator is characterised by a probability density function the probability that the point estimate actually equals the true value of the parameter is zero. Using the spring scales analogy, we see that no matter how accurate and precise the scales are, the probability of obtaining the exact weight (with arbitrary large number of digits) is zero. We need, therefore, to attach some measure of the possible error of the estimate to the point estimate. For that purpose, we attempt to determine an interval, called confidence interval,

containing the true parameter value θ with a given probability 1– α, the so-called confidence level:

P ( t n , 1 ( x ) < θ < t n , 2 ( x ) ) = 1 − α , 3.1

where α is a confidence risk. The endpoints of the interval (also known as confidence limits), depend on the available sample and are determined taking into account the sampling distribution:

We have assumed that the interval endpoints are finite, the so-called two-sided (or two-tail) interval estimation. Sometimes we will also use one-sided (or one-

tail) interval estimation by setting t n , 1 ( x ) = −∞ or t n , 2 ( x ) = +∞ .

Let us now apply these ideas to the spring scales example. Imagine that, as happens with unbiased point estimators, there were no systematic error and furthermore the measured errors follow a known normal distribution; therefore, the measurement error is a one-dimensional random variable distributed as N 0, σ , with known σ. In other words, the distribution function of the random weight variable,

W, is F W ( w ) ≡ F ( w ) = N ω , σ ( w ) . We are now able to determine the two-sided 95% confidence interval of ω, given a measurement w, by first noticing, from the normal distribution tables, that the percentile 97.5% (i.e., 100– α/2, with α in percentage) corresponds to 1.96 σ:

Thus:

F ( w ) = 0 . 975 ⇒ w 0 . 975 = 1 . 96 σ . 3.2

Given the symmetry of the normal distribution, we have:

P ( w < ω + 1 . 96 σ ) = 0 . 975 ⇒ P ( ω − 1 . 96 σ < w < ω + 1 . 96 σ ) = 0 . 95 ,

leading to the following 95% confidence interval:

ω − 1 . 96 σ < w < ω + 1 . 96 σ . 3.3

Hence, we expect that in a long run of measurements 95% of them will be inside the ω ± 1.96σ interval, as shown in Figure 3.2a. Note that the inequalities 3.3 can also be written as:

84 3 Estimating Data Parameters

w − 1 . 96 σ < ω < w + 1 . 96 σ , 3.4

allowing us to define the 95% confidence interval for the unknown weight (parameter) ω given a particular measurement w. (Comparing with expression 3.1 we see that in this case θ is the parameter ω, t 1,1 = w – 1.96 σ and t 1,2 = w + 1.96 σ.)

As shown in Figure 3.2b, the equivalent interpretation is that in a long run of measurements, 95% of the w ± 1.96 σ intervals will cover the true and unknown weight ω and the remaining 5% will miss it.

Figure 3.2. Two interpretations of the confidence interval: a) A certain percentage of the w measurements (#1,…, #10) is inside the ω ± 1.96σ interval; b) A certain percentage of the w ± 1.96 σ intervals contains the true value ω.

Note that when we say that the 95% confidence interval of ω is w ± 1.96σ, it does not mean that the probability that “ ω falls in the confidence interval is 95% . ” This is a misleading formulation since ω is not a random variable but an unknown parameter. In fact, it is the confidence interval endpoints that are random variables.

For an arbitrary risk, α, we compute from the standardised normal distribution the 1– α/2 percentile:

0 , 1 ( z ) = 1 − α / 2 ⇒ z 1 − α / 2 . 3.5

We now use this percentile in order to establish the confidence interval:

w − z 1 − α / 2 σ < ω < w + z 1 − α / 2 σ . 3.6

The factor z 1 − α2 / σ is designated as tolerance, ε, and is often expressed as a

percentage of the measured value w, i.e., ε = 100 z 1 − α2 / σ / w %.

It is customary to denote the values obtained with the standardised normal distribution by the letter z, the so called z-scores.

3.2 Estimating a Mean

In Chapter 1, section 1.5, we introduced the notions of confidence level and interval estimates, in order to illustrate the special nature of statistical statements and to advise taking precautions when interpreting them. We will now proceed to apply these concepts to several descriptive statistics that were presented in the previous chapter.