How the L1 Penalty in Lasso Regression Forces Coefficients to Zero for Feature Selection

In 1996, statistician Robert Tibshirani published a 22-page paper in the Journal of the Royal Statistical Society that fundamentally altered how algorithms process high-dimensional data. By introducing the Least Absolute Shrinkage and Selection Operator (Lasso), Tibshirani provided a mathematical mechanism to not just shrink regression coefficients, but force the multipliers of less important variables exactly to zero.[1][5]

Before this publication, models facing hundreds of variables relied heavily on Ridge regression or stepwise selection. Ridge regression applies an L2 penalty—squaring the coefficients—which shrinks them toward zero to prevent overfitting, but never actually removes them from the equation. A model trained on 1,000 inputs using an L2 penalty still requires all 1,000 inputs to make a single prediction.[4][5]

Lasso replaced the squared L2 penalty with an L1 penalty, which takes the absolute value of the coefficients. This algebraic shift alters the geometry of the optimization problem. When the algorithm minimizes the mean squared error alongside this L1 penalty, the constraint space forms a diamond with sharp corners.[1][3][5]

Because the elliptical contours of the error function expand outward from the unpenalized solution, they almost always hit the sharp corners of the L1 diamond first. Those corners lie exactly on the axes of the coordinate system. When the intersection occurs on an axis, the corresponding coefficient is set to precisely 0.[1][5][6]

This geometric intersection gives Lasso its dual capability: it regularizes the model to prevent overfitting while simultaneously performing feature selection. If a dataset contains 50,000 features but only 15 genuinely drive the outcome, the L1 penalty drops the other 49,985 by zeroing out their weights. This allows ordinary least squares to function even when the number of predictors exceeds the number of observations.[2][3][5]

The strength of this penalty is controlled by a tuning parameter, lambda (λ). When λ is set to 0, the penalty vanishes entirely, and the model reverts to standard linear regression. As λ increases, the diamond shrinks, forcing more coefficients to 0 until the model is entirely empty.[3][5]

Data scientists typically use cross-validation to find the optimal λ—the point where the model drops enough noise to generalize well to new data, but retains enough signal to remain accurate. The scikit-learn library in Python implements this optimization efficiently using coordinate descent algorithms, allowing models to converge rapidly even on massive datasets.[2][3]

However, the L1 penalty has a documented mathematical weakness when dealing with highly correlated variables. If three features contain nearly identical information, Lasso tends to arbitrarily select one and zero out the other two. This instability makes the resulting model harder to interpret, as the chosen variable might change entirely if the training data shifts slightly.[2][4]

To resolve this instability, statisticians Hui Zou and Trevor Hastie introduced the Elastic Net in April 2005, publishing their findings across 20 pages. As Zou and Hastie wrote in their introduction, the method "encourages a grouping effect, where strongly correlated predictors tend to be in or out of the model together." By combining both the L1 penalty of Lasso and the L2 penalty of Ridge regression, Elastic Net retains Lasso's ability to drop useless variables while inheriting Ridge's stability.[4]

The choice between these penalties now forms the foundation of regularized regression. The L1 penalty remains the definitive tool for pure feature selection, isolating the mathematical signal in datasets that are otherwise too wide to parse. The next frontier in regularization involves adapting these exact geometric constraints for deep neural networks, where enforcing sparsity could drastically reduce the billions of parameters currently required for large language models.[2][5][6]