User:Sbowen209/sandbox

In statistics, the LASSO (Least Absolute Shrinkage and Selection Operator), is a regularized regression method that is used for variable shrinkage and selection in linear or logistic regression models.

History
The LASSO method was created by Robert Tibshirani after being influenced by Leo Breiman's work on subset selection. The LASSO method has grown in popularity due to the importance of variable selection in big data problems and advances in computational technologies.

The LASSO
Given a linear regression with a response variable $$ y \in R^n $$ and a matrix $$ X \in R^{nxp} $$ of explanatory variables, the LASSO method produces the estimate for the unknown parameter $$ \beta$$:


 * $$ \hat{\beta}^{lasso} = \underset{\beta}{\operatorname{argmin}} (\| y-X \beta \|^2) \mbox{ s.t. } \sum_{j=1}^p |\beta_j| \leq t.$$

This is equivalent to the following loss function:
 * $$ \hat{\beta}^{lasso} = \underset{\beta}{\operatorname{argmin}} (\| y-X \beta \|^2 + \lambda\|\beta\|_1) ,$$

where $$\|\beta\|_1 $$ is an L1 penalty and the tuning parameter $$\lambda$$ has a one-to-one correspondence with $$ t $$ and governs the strength of this penalty. Like other types of regularization, such as ridge regression, the size of the coefficients shrink as $$\lambda$$ is increased. However, unlike ridge regression, some of the coefficients are shrunken identically to zero. In this manner, the LASSO performs continuous variable selection.

Elastic Net Regularization
The LASSO may saturate quickly in the case of data with a large number of predictors but only few observations. Additionally, it will ignore variables if they are highly correlated to another. Variations such as the elastic net method can overcome these limitations by employing penalties from both the LASSO and ridge regression.

Computation
Because the LASSO utilizes an absolute value operation in its penalty, the loss function is not differentiable and in general does not have a closed form. Therefore, convex optimization algorithms are required to find a solution. In the original paper, solutions were found through quadratic programming, but more recent methods such as least-angle regression and coordinate descent have proven more efficient.

Software
"Glmnet: Lasso and elastic-net regularized generalized linear models" is software which is implemented as an R source package. This includes fast algorithms for estimation of generalized linear models with ℓ1 (the lasso), ℓ2 (ridge regression) and mixtures of the two penalties (the elastic net) using cyclical coordinate descent, computed along a regularization path.