Multiple Regression Analysis
13.4 Multiple Regression Analysis
In multiple regression, the objective is to build a probabilistic model that relates a dependent variable y to more than one independent or predictor variable. Let k repre- sent the number of predictor variables (k 2) and denote these predictors by
x 1 ,x 2 , c, x . For example, in attempting to predict the selling price of a house, we k might have k53 with x 1 5 size (ft 2 ), x 2 5 age (years) , and x 3 5 number of rooms .
DEFINITION
The general additive multiple regression model equation is
(13.15) where E(P) 5 0 and V(P) 5 s 2 . In addition, for purposes of testing hypotheses
Y5b 0 1b 1 x 1 1b 2 x 2 1c1b k x k 1P
and calculating CIs or PIs, it is assumed that is normally distributed. P
Let x 1 ,x , c, x 2 k
be particular values of x 1 , c, x k . Then (13.15) implies that m Yx 1 ,c,x k 5b 0 1b 1 x 1 1c1b k x k (13.16) Thus just as b 0 1b 1 x describes the mean Y value as a function of x in simple linear
regression, the true (or population) regression function b 0 1b 1 x 1 1c1b k x k gives the expected value of Y as a function of x 1 , c, x k . The b i ’s are the true (or population) regression coefficients. The regression coefficient b 1 is interpreted as the expected change in Y associated with a 1-unit increase in x 1 while x 2 , c, x k are
held fixed. Analogous interpretations hold for b 2 , c, b k .
Models with Interaction and Quadratic Predictors
If an investigator has obtained observations on y, x 1 , and x 2 , one possible model is Y5b 0 1b 1 x 1 1b 2 x 2 1P . However, other models can be constructed by forming predictors that are mathematical functions of x 1 andor x 2 . For example, with x 3 5x 1 2
and x 4 5x 1 x 2 , the model
Y5b 0 1b 1 x 1 1b 2 x 2 1b 3 x 3 1b 4 x 4 1P
has the general form of (13.15). In general, it is not only permissible for some pre- dictors to be mathematical functions of others but also often highly desirable in the sense that the resulting model may be much more successful in explaining variation in y than any model without such predictors. This discussion also shows that polyno- mial regression is indeed a special case of multiple regression. For example, the quad-
ratic model Y5b 0 1b 1 x1b 2 x 2 1P has the form of (13.15) with k 5 2, x 1 5x ,
and . x 2 5x 2
For the case of two independent variables, x 1 and x 2 , consider the following
four derived models.
1. The first-order model: Y5b 0 1b 1 x 1 1b 2 x 2 1P
2. The second-order no-interaction model:
Y5b 0 1b 1 x 1 1b 2 x 2 1b 3 x 1 2 1b 4 x 2 1P
CHAPTER 13 Nonlinear and Multiple Regression
3. The model with first-order predictors and interaction: Y5b 0 1b 1 x 1 1b 2 x 2 1b 3 x 1 x 2 1P
4. The complete second-order or full quadratic model:
Understanding the differences among these models is an important first step in building realistic regression models from the independent variables under study.
The first-order model is the most straightforward generalization of simple linear regression. It states that for a fixed value of either variable, the expected value of Y is a linear function of the other variable and that the expected change in Y asso-
ciated with a unit increase in x 1 (x 2 ) is b 1 (b 2 ) independent of the level of x 2 (x 1 ) . Thus if we graph the regression function as a function of x 1 for several different val-
ues of x 2 , we obtain as contours of the regression function a collection of parallel
lines, as pictured in Figure 13.13(a). The function y5b 0 1b 1 x 1 1b 2 x 2 specifies
a plane in three-dimensional space; the first-order model says that each observed value of the dependent variable corresponds to a point which deviates vertically from
this plane by a random amount . P
According to the second-order no-interaction model, if x 2 is fixed, the
expected change in Y for a 1-unit increase in x 1 is
2 (b 0 1b 1 x 1 1b 2 x 2 1b 3 x 2 1 1b 4 x 2 )5b 1 1b 3 1 2b 3 x 1 Because this expected change does not depend on x 2 , the contours of the regression
function for different values of x 2 are still parallel to one another. However, the dependence of the expected change on the value of x 1 means that the contours are
now curves rather than straight lines. This is pictured in Figure 13.13(b). In this case, the regression surface is no longer a plane in three-dimensional space but is instead
a curved surface.
The contours of the regression function for the first-order interaction model
are nonparallel straight lines. This is because the expected change in Y when x 1 is
increased by 1 is
b 0 1b 1 (x 1 1 1) 1 b 2 x 2 1b 3 (x 1 1 1)x 2
2 (b 0 1b 1 x 1 1b 2 x 2 1b 3 x 1 x 2 )5b 1 1b 3 x 2
This expected change depends on the value of x 2 , so each contour line must
have a different slope, as in Figure 13.13(c). The word interaction reflects the fact that an expected change in Y when one variable increases in value depends on the value of the other variable.
Finally, for the complete second-order model, the expected change in Y when
x 2 is held fixed while x 1 is increased by 1 unit is b 1 1b 3 1 2b 3 x 1 1b 5 x 2 , which is
a function of both x 1 and x 2 . This implies that the contours of the regression function
are both curved and not parallel to one another, as illustrated in Figure 13.13(d).
Similar considerations apply to models constructed from more than two inde- pendent variables. In general, the presence of interaction terms in the model implies that the expected change in Y depends not only on the variable being increased or decreased but also on the values of some of the fixed variables. As in ANOVA, it is
possible to have higher-way interaction terms (e.g., x 1 x 2 x 3 ), making model inter-
pretation more difficult.
13.4 Multiple Regression Analysis
1 .5x 1 .25x 1 2 x 2 .5x 2 2
1 .5x 1 .25x 2 1 x 2 .5x 2 2 x 1 x 2
Figure 13.13 Contours of four different regression functions
Note that if the model contains interaction or quadratic predictors, the generic interpretation of a b i given previously will not usually apply. This is because it is not then possible to increase x i by 1 unit and hold the values of all other predictors fixed.