Matrix Approach to Simple Linear Regression

Matrix

• Collection of elements arranged in rows and columns

• Elements will be numbers or symbols

• For example:

• Rows denoted with the i subscript

• Columns denoted with the j subscript

• The element in row 1 col 2 is 3

• The element in row 3 col 1 is 2

• Elements often expressed using symbols

• Matrix A has r rows and c columns

• Said to be of dimension r × c

• Element is in i^th row and j^th col

• A matrix is square if r = c

• Called a column vector if c = 1

• Called a row vector if r = 1

Matrix Operations

• Transpose
– Denoted as A'
Row 1 becomes Col 1, Row r becomes Col r

Col 1 becomes Row 1, Col c becomes Row c
– If A = [] then A' = []
– If A is r × c then A' is c × r

• Addition and Subtraction
– Matrices must have the same dimension
– Addition/ subtraction done on element by element basis

• Multiplication
– If scalar then
– If multiplying two matrices (C = AB)

Cols of A must equal Rows of B
Resulting matrix of dimension Rows(A) × Col(B)
– Elements obtained by taking cross products of rows of A with cols of
B

Regression Matrices

• Consider example with n = 4

• Consider ex pressing observations :

Special Regression Examples

• Using multiplication and transpose

• Will use these to compute etc.

Special Types of Matrices

• Symmetric matrix
– When A = A'
– Requires A to be square
– Example: X'X

• Diagonal matrix
– Square matrix with o®-diagonals equal to zero
– Important example: Identity matrix

– IA = AI = A

Linear Dependence

• Consider the matrix

the columns of Q are vectors.

• If there is a relationship between the columns of a matrix
such that

and not all ¸ are 0, then the set of column vectors are
linearly dependent .
– For the above example,

• If such a relationship does not exist then the set of columns
are linearly independent.

• Similarly consider rows

Rank of a Matrix

• The rank of a matrix is number of linear independent columns
(or rows)

• Rank of a matrix cannot exceed min(r; c)

• Full Rank ´ all columns are linearly independent

• Example:

– The rank of Q is 2

Inverse of a Matrix

• Inverse similar to the reciprocal of a scalar

• Inverse defined for square matrix of full rank

• Want to find the inverse of S, such that

• Easy example: Diagonal matrix
– Let then

inverse of each element
on the diagonal

• General procedure for 2 × 2 matrix

• Consider:

1. Calculate the determinant D = a ٠ d - b ٠ c
If D = 0 then the matrix has no inverse.
2. In , switch a and d; make c and b negative ; multiply each element
by

– Steps work only for a 2 × 2 matrix.
– Algorithm for 3 × 3 given in book