Linear Regression

8.2 Linear Regression

We perform linear regression with the method of least squares. With this method, we compute the coefficients (slope) and (y-intercept) of the straight line equation m b

y = mx + b (8.4) such that the sum of the squares of the errors will be minimum. We derive the values of and , m b

that will make the equation of the straight line to best fit the observed data, as follows: Let and be two related variables, and assume that corresponding to the values x y

x 1 ,,,, x 2 x 3 …x n , we have observed the values y 1 ,,,, y 2 y 3 …y n . Now, let us suppose that we have plotted the values of versus the corresponding values of , and we have observed that the y x

points ( x 1 , y 1 )x , ( 2 , y 2 )x , ( 3 , y 3 )…x ,, ( n , y n ) approximate a straight line. We denote the straight line equations passing through these points as

y 1 = mx 1 + b y 2 = mx 2 + b

y 3 = mx 3 + b (8.5)

… y n = mx n + b

In (8.5), the slope and y-intercept are the same in all equations since we have assumed that m b all points lie close to one straight line. However, we need to determine the values of the unknowns

m and from all equations; we will not obtain valid values for all points if we solve just two b n

8 − 2 Numerical Analysis Using MATLAB ® and Excel ® , Third Edition Copyright © Orchard Publications

Linear Regression

equations with two unknowns. * The error (difference) between the observed value y 1 , and the value that lies on the straight line,

is y 1 – ( mx 1 + b ) . This difference could be positive or negative, depending on the position of the observed value, and the value at the point on the straight line. Likewise, the error between the

observed value y 2 and the value that lies on the straight line is y 2 – ( mx 2 + b ) and so on. The straight line that we choose must be a straight line such that the distances between the observed

values, and the corresponding values on the straight line, will be minimum. This will be achieved if we use the magnitudes (absolute values) of the distances; if we were to combine positive and negative values, some may cancel each other and give us an erroneous sum of the distances. Accordingly, we find the sum of the squared distances between observed points and the points on the straight line. For this reason, this method is referred to as the method of least squares.

Let the sum of the squares of the errors be

2 squares 2 = [ ∑ y

squares is a function of two variables and , to minimize (8.6) we must equate to m ∑ b

zero its two partial derivatives with respect to and . Then, m b

∂ squares = – 2x 1 [ y 1 – ( mx 1 + b ) ] – 2x 2 [ y 2 – ( mx + b ) ]… –

squares = – 2y [ 1 – ( mx 1 + b ) ] – 2y [ 2 – ( mx 2 + b ) ]… –

– 2y [ n – ( mx n + b ) ] = 0

The second derivatives of (8.7) and (8.8) are positive and thus

squares ∑ will have its minimum

value. Collecting like terms, and simplifying (8.7) and (8.8) we obtain

* A linear system of independent equations that has more equations than unknowns is said to be overdetermined and no exact solution exists. On the contrary, a system that has more unknowns than equations is said to be underdetermined and these systems have infinite solutions.

Numerical Analysis Using MATLAB ® and Excel ® , Third Edition

Copyright © Orchard Publications