The Method of Least Squares

The Method of Least Squares is a method for fitting an analytical function to data. The
data may for example be gathered from experiments. As you can imagine, Least Squares is
an extremely useful method. In this lab you will learn how to find the (i) linear function
y = mx + b and (ii) the quadratic function y = ax^2 + bx + c that are the \best" linear
and quadratic fits, respectively, to a set of data points {(x_i, y_i) ∈R^2, fori = 1,…N}. To
quantify the term "best fit" we minimize the error between the function that is being fitted
and the data points. So we have to solve an optimization problem. In general, even if know
from theory that the data should be modeled by a linear or quadratic function, noise in
the physical process used to generate the data and inevitable errors in the measurement of
the data mean that the data points will not exactly lie on a line or a quadratic. In other
words, the error between the between the function that is being fitted and the data points
will probably not be zero . Nevertheless, for a linear t, for example, we want to find the
values of slope m and y - intercept b that make the error as small as possible.

Fitting a Linear Function to Data

Given a set of data points {(x_i, y_i) ∈R^2, fori = 1,…N} and a function y = f(x), we define
the mean square error, E, to be the average of the squares of the vertical distances between
data and the y-values of the function,
(1)

In the special case of a linear function y = f(x) = mx+b, the error E depends on the values
of m and b, and Equation 1 becomes
(2)

So it makes sense to de fine the \best- t" line to be the one whose parameters (m, b) minimize
the error E(m, b).

(1) Your first task is to find the critical points of the function E(m, b) of two variables (m, b).
Begin by showing that the equations
(3)

can be ex pressed as
(4)
(5)

Notice that this is a system of two linear equations in two unknkowns (m, b). [Remember
that the x_i and y_i are numbers !]

(2) Download the data le Data1.dat from the course web page and plot it in Matlab. For
this you will need to use the load commands:

Data = load('Data1.dat');
x = Data(:,1)
y = Data(:,2)
plot(x,y,'bx');

Now write a Matlab program to find the line that best fits the data. This will require
you to solve the system of linear equations (4) and (5). To set up the system, you may
find the Matlab functions size and sum helpful . If you set it up as a 2 * 2 matrix system,
Au = v where u is the vector of unknowns, then you can solve the equation in Matlab using
the command u=A\v, which means u = A^-1v. For the data in Data1.dat, what are the best
values of m and b? Plot the data and the best t line on the same axes.

Fitting a Quadratic Function to Data

(3) Download the data le Data2.dat from the course web page and plot it in Matlab. As
you can see this data looks like it might t a quadratic function y = f(x) = ax^2 + bx + c
quite well.

In ana logy with (1) above, write down an error function E(a, b, c) to be minimized and
corresponding equations for the critical points. This should give you a 3 * 3 system of linear
equations in unknowns (a, b, c).

Use these equations to write a Matlab program to find the quadratic function that best
fits data. Test you program on the data set Data2.dat. What are the best values of (a, b, c)?
Plot the data and the best t quadratic on the same axes.