User:BenFrantzDale/Matrix decomposition

I don't like how I was taught eigendecomposition and SVD; it never made sense. Here's a stab at how I would do it...

Motivation: Linear algebra is useful for a startling number of applications...

[Describe the rules of matrix multiplication.]

Diagonal matrices
A diagonal matrix is one that has all its non-zero entries on the diagonal. That is,
 * $$M_{ij}=0 \forall i\neq j$$.

This has a very practical meaning: It scales things only in axial directions. This is a little like the ellipse-select tool in Photoshop: you can make it wider (x) or taller (y) but you can't make it long and skinny along a diagonal. That is intuitively what it means for a matrix to be diagonal.

Rotation matrices
A rotation matrix is one that rotates vectors about the origin without scaling them. These matrices have a number of interesting properties:


 * 1) The columns are all orthogonal.
 * 2) The columns all have norm 1.
 * 3) $$R^\top = R^{-1}$$.
 * 4) The rows are all orthogonal.
 * 5) The rows all have norm 1.

If a matrix rigidly rotates things, then for all orthogonal pairs of vectors, u, and v, they should still be orthogonal after rotation. That is: $$u\cdot v = 0 = (Ru)\cdot(Rv)$$ Suppose R has two non-orthogonal columns, column i and j. Then let u be all zeros except for the ith entry is 1 and v all zeros except the jth entry is 1. Clearly $$u\cdot v=0$$. Now Ru and Rv are just the ith and jth columns of R respectively, so by contradiction, if the columns were not orthogonal, we could always find orthogonal pairs of vectors that weren't rigidly rotated by R.

For 2, if a matrix doesn't scale vectors, only redirects them, then
 * $$\|Rv\| = \|v\| \forall v$$

Suppose c is the ith column of R and has norm not equal to 1. Then let u be the same vector, all zeros except the ith element is 1. Clearly $$\|u\|=1$$. But then
 * $$Ru=c$$ and so $$\|Ru\|=\|c\|\neq \|u\|=1$$.

So by contradiction, all columns must have norm 1.

Three is interesting: The transpose is the inverse. Consider again u being all zeros except for a 1 in the ith entry. Then Ru=c is the ith column of R. What should $$R^{-1}$$ look like? Well, to get $$u=R^{-1}Ru=R^{-1}c$$,the ith row of $$R^{-1}$$ will have to dot with c to get 1. The only vector that will do that is c (which, as we established, has a norm of 1). All of the other rows must be orthogonal to c, and this has to hold for all i. Clearly $$R^\top$$ has the property that the ith row equals the ith column of R. Similarly the other rows of $$R^\top$$ are the other columns of R so we know they are orthogonal!

Four and five follow from one through three: if the transpose is a rotation matrix, then any column properties apply to the rows too.

Symmetric matrices
Symmetric matrices are matrices for which $$M_{ij}=M_{ji}$$. These are extraordinarily common in science and engineering and have lots of nice properties. But aside from an array of numbers on a page, what does that mean?

Symmetric positive (semi-)definite matrices
Let's start with positive semi-definite matrices. These are those symmetric matrices that also have the following obtuse property:
 * $$v^\top M v \geq 0 \forall v$$.

What this really means is that M describes an oriented ellipsoid. Why isn't obvious.

Suppose there's a matrix, A, that scales and possibly turns or squashes v. Given a v, we could ask how much A changes its length. That is, if v is a unit vector, what is $$\|Av\|$$? Well, it's just
 * $$\sqrt{(Av)^\top Av} = \sqrt{v^\top A^\top A v}$$.