Linear Algebra

Square Matrices

First, define 0_n as the n × n matrix containing all zeros and I _n as the square matrix
(n × n) with ones on the diagonal and zeros elsewhere. I_n is the identity matrix, and the
subscript for dimension is often sup pressed .

Note that A + 0 = A (as suming all are n × n and that A0 = 0 so 0 behaves like a zero
should.

Note also that AI = IA = A.

Thus, in terms of transforms , I corresponds to the linear transform from n space to itself
which just leaves each vector alone, that is, L(x) = x. 0 corresponds to the transform
which maps all n vectors into an n vector of all zeros.

Finally, if x = 0 (a column vector of all zeros), then we see that Mx = 0.

Are there other, non-zero, vectors that get mapped into zero? This turns out to be critical.
Note that if Mx = 0 for non-zero x, then the elements of x give a non-trivial linear
combination of the rows of M which add to zero , so the rows of M are linearly dependent .

If no non-zero x gets mapped to zero, then the rows of M are linearly independent.

We define the maximal number of linearly independent rows of M as its row rank and
similarly for the column rank. It is easy to prove that the row and column ranks of a matrix
are the same, so we can just talk of the rank of a matrix. If it is n × n its rank can be no
bigger than n; if it is n, then we say it has full rank, otherwise it is less than full rank.

Inverses of square matrices

Define the inverse of A, A^-1 such that AA^-1 = A^-1A = I. Inverse pop up all the
time; question is how to find them and do they exist.

The computation is tedious. But we can understand by looking at the transform
corresponding to A and it’s inverse.
(Get computation by just solving for A ^-1 in the matrix equation AA ^-1 = I, nothing
difficult, just tedious and not very insightful.

What is the inverse of a transform? It undoes the transform, that is, if we do a transform and
then its inverse, we get back to the original vector. Note that the product of the matrix and
its inverse is the identity matrix.

A necessary condition for the inverse to exist is that it be one to one and onto.

One to one means that each vector in the range of the transform is the transform of only one
vector, that is, we do not have L(x) = L(y) unless x = y. We need this since transforms
take a vector into only one vector, and since the inverse would be a transform, it too must
take a vector into only one vector. But if L(x) = L(y), what is the inverse of L(x); is it x
or y?

Onto means that every vector in the range space is mapped by something, so that there is an
x to make y = L(x) for every y. If the transform is not onto, then some vectors in the range
space have no map from the domain, so how do we then invert?

Note that L(0) = 0 since L(0) = L(x − x) = L(x) − L(x) = 0.

If any other points in the domain get mapped into zero, then the transform is not one to one,
and so inverse does not exist.
If nothing other than 0 gets mapped into 0, then it is easy to show that the transform is one
to one.
(Suppose not. Then we would have L(x) = L(z) = y, x ≠ z, and so L(x − z) = 0 but
x − z ≠ 0.)

Thus to see if inverses exist, we need merely look at what gets mapped into zero (the kernel
of the transform) and see if only zero gets mapped into zero.

Thus consider the transform T(x₁, x₂) = (x₁) which takes R² → R¹. Clearly all
elements (0, x₂) get mapped to 0, so the transform is not one to one and hence inverse does
not exist.

Consider mapping R² → R² by Note that L(0) = 0. Can
L(x) = 0, x ≠ 0? If so, both x₁ + x₂ = 0, x₁ − x₂ = 0. Solve and you will see that
this holds only for x₁ = x₂ = −. (Again, note the tie between matrices, linear transforms
and solving linear equations.)

First, only square matrices can have inverses. If M is not square, it maps Rⁿ → R^m. If
m > n, then more elements the transform cannot be onto (only onto transform could be
onto an n dimensional subspace. If m < n, this discussion holds for the inverse.

So only square matrices have inverses. All we have to do is to check that only 0 gets mapped
to 0.

But as we have seen, if something else gets mapped to zero, the columns of the matrix are
linearly dependent.

Thus a square matrix has an inverse if it is full rank.

Determinant

Determinants are usually treated as annoying things to compute. But the determinant of A,
(|A|), is simply the hypervolume of the hyperrectangle defined by either the rows or columns
of A.

Note that is one row or column is a linear combination of the others, then the hyperrectangle
is not of full dimension, that is, has hypervolume of zero.

Thus a matrix has an inverse if and only if its determinant is non-zero.

Quadratic forms

Letting x be a vector and A a conformable matrix, we often deal with quadratic forms x'Ax
which is a scalar (x' is 1 × n, so product is 1 × 1, that is, a scalar).

A matrix is positive definite (PD) if all quadratic forms involving it are positive (other than
the trivial x = 0). Similarly for negative definite ; positive semi-definite has all quadratic
forms involving A being non-negative and similarly for negative semi-definite.

Note that quadratic forms are the matrix ana logue of scalar quadratic equations. To see this,
let Then

(7)

Matrix calculus

We need matrix calculus to do minima and maxima; matrix calculus is taking derivatives with
respect to a vector x; this is defined by the taking the derivatives with respect to each
element of x and then treating all the derivatives as a vector.

Thus this is purely a notation saving device and the proofs are trivial.

(8)
(9)