User:Bruguiea/sandbox

Playground
$$y = X \beta + \epsilon$$

$$(n; 1)$$

$$(n; p)$$

$$(p; 1)$$

$$\hat{\beta} = (X' X) ^{-1} X y$$

$$\hat{\epsilon} =\left(I - (X' X) ^{-1} X \right) y$$

$$n - r$$

$$t_{n-r} = \frac{\hat{\beta _i}}{\hat{\epsilon} ' \hat{\epsilon}} = \frac{\hat{\beta _i}}{y' \left(I - (X' X) ^{-1} X \right)' \left(I - (X' X) ^{-1} X \right) y}$$

$$t_{n-r} \rightarrow p$$

$$U = \operatorname{ER} - \alpha \operatorname{Risk}$$

Special case of covariance matrices
A covariance matrix $$\Sigma$$ can be represented as the product $$A' A$$. Its eigenvalues are positive:


 * $$\Sigma v_i = \lambda _i v_i$$


 * $$A' A v_i = \lambda _i v_i$$


 * $$v_i' A' A v_i = v_i' \lambda _i v_i$$


 * $$ \left\| A v_i \right\| = \lambda _i \left\| v_i \right\|$$


 * $$ \lambda _i = \frac{\left\| v_i \right\|}{\left\| A v_i \right\|} \geq 0$$

The eigenvectors are orthogonal to one another:


 * $$A' A v_i = \lambda _i v_i$$


 * $$v _j' A' A v_i = \lambda _i v_j' v_i$$


 * $$(A' A v_j )' v_i = \lambda _i v_j' v_i$$


 * $$\lambda _j v_j ' v_i = \lambda _i v_j' v_i$$


 * $$(\lambda _j - \lambda _i) v_j ' v_i = 0$$


 * $$v_j ' v_i = 0$$ (different eigenvalues, in case of multiplicity, the basis can be orthogonalized)

The Rayleigh quotient can be expressed as a function of the eigenvalues by decomposing any vector $$x$$ on the basis of eigenvectors:
 * $$x = \sum _{i=1} ^n \alpha _i v_i$$


 * $$\rho = \frac{x' A' A x}{x' x}$$


 * $$\rho = \frac{(\sum _{j=1} ^n \alpha _j v_j)' A' A (\sum _{i=1} ^n \alpha _i v_i)}{(\sum _{j=1} ^n \alpha _j v_j)' (\sum _{i=1} ^n \alpha _i v_i)}$$

Which, by orthogonality of the eigenvectors, becomes:


 * $$\rho = \frac{\sum _{i=1} ^n \alpha _i ^2 \lambda _i}{\sum _{i=1} ^n \alpha _i ^2}$$

If a vector $$x$$ maximizes $$\rho$$, then any vector $$k. x$$ (for $$k \ne 0$$) also maximizes it, one can reduce to the Lagrange problem of maximizing $$\sum _{i=1} ^n \alpha _i ^2 \lambda _i$$ under the constrainst that $$\sum _{i=1} ^n \alpha _i ^2 = 1$$.

Since all the eingenvalues are non-negative, the problem is convex and the maximum occurs on the edges of the domain, namely when $$\alpha _1 = 1$$ and $$\forall i > 1 \alpha _i = 0$$ (when the eigenvalues are ordered in decreasing magnitude).

This property is the basis for principal components analysis and canonical correlation.