1

I am confused about the behavior of the lasso path when running linear_model.lars_path(model = 'lasso') in scikit-learn.

I thought that once a weight (coefficient) becomes active (diff from 0) it must remain active during all the forthcoming steps of the LARS algorithm.

When running the algorithm on my data, I noticed that sometimes a coefficient would become active then later it would go to zero (be removed from the active set). It this the correct behavior of the LARS algorithm, or could there be a bug in the scikit-learn implementation?.

hjpotter92
  • 78,589
  • 36
  • 144
  • 183
user2043236
  • 71
  • 1
  • 4
  • 2
    yes, that is possible. Maybe look at the book Elements of Statistical Learning for more details on path algorithms in linear models. – Andreas Mueller Feb 05 '13 at 20:20

1 Answers1

1

This is correct behavior for the L1-regularized version of LARS (and L1-regularized regression is generally known as "lasso").

In the L1 version, if a step along the LARS path causes the sign of the correlation of a column of the data matrix with the residual to be different from the sign of the corresponding regression coefficient (e.g. sgn(x_i^\top(y-X\beta)) != sgn(\beta_i) ), then this column/coefficient will be removed from the active path. (You can find the original description in the 2003 paper "Least Angle Regression" by Efron et al.)

In contrast, the active set grows at each iteration in the L0-regularized version of LARS.

user1149913
  • 4,463
  • 1
  • 23
  • 28