ConvergenceWarning: lbfgs failed to converge (status=1): STOP: TOTAL NO. of ITERATIONS REACHED LIMIT

Question

I have a dataset consisting of both numeric and categorical data and I want to predict adverse outcomes for patients based on their medical characteristics. I defined a prediction pipeline for my dataset like so:

X = dataset.drop(columns=['target'])
y = dataset['target']

# define categorical and numeric transformers
numeric_transformer = Pipeline(steps=[
    ('knnImputer', KNNImputer(n_neighbors=2, weights="uniform")),
    ('scaler', StandardScaler())])

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))])

#  dispatch object columns to the categorical_transformer and remaining columns to numerical_transformer
preprocessor = ColumnTransformer(transformers=[
    ('num', numeric_transformer, selector(dtype_exclude="object")),
    ('cat', categorical_transformer, selector(dtype_include="object"))
])

# Append classifier to preprocessing pipeline.
# Now we have a full prediction pipeline.
clf = Pipeline(steps=[('preprocessor', preprocessor),
                      ('classifier', LogisticRegression())])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

clf.fit(X_train, y_train)
print("model score: %.3f" % clf.score(X_test, y_test))

However, when running this code, I get the following warning message:

ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

    model score: 0.988

Can someone explain to me what this warning means? I am new to machine learning so am a little lost as to what I can do to improve the prediction model. As you can see from the numeric_transformer, I scaled the data through standardisation. I am also confused as to how the model score is quite high and whether this is a good or bad thing.

score 171 · Accepted Answer · edited May 08 '21 at 19:56

The warning means what it mainly says: Suggestions to try to make the solver (the algorithm) converges.

lbfgs stand for: "Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm". It is one of the solvers' algorithms provided by Scikit-Learn Library.

The term limited-memory simply means it stores only a few vectors that represent the gradients approximation implicitly.

It has better convergence on relatively small datasets.

But what is algorithm convergence?

In simple words. If the error of solving is ranging within very small range (i.e., it is almost not changing), then that means the algorithm reached the solution (not necessary to be the best solution as it might be stuck at what so-called "local Optima").

On the other hand, if the error is varying noticeably (even if the error is relatively small [like in your case the score was good], but rather the differences between the errors per iteration is greater than some tolerance) then we say the algorithm did not converge.

Now, you need to know that Scikit-Learn API sometimes provides the user the option to specify the maximum number of iterations the algorithm should take while it's searching for the solution in an iterative manner:

LogisticRegression(... solver='lbfgs', max_iter=100 ...)

As you can see, the default solver in LogisticRegression is 'lbfgs' and the maximum number of iterations is 100 by default.

Final words, please, however, note that increasing the maximum number of iterations does not necessarily guarantee convergence, but it certainly helps!

Update:

Based on your comment below, some tips to try (out of many) that might help the algorithm to converge are:

Increase the number of iterations: As in this answer;
Try a different optimizer: Look here;
Scale your data: Look here;
Add engineered features: Look here;
Data pre-processing: Look here - use case and here;
Add more data: Look here.

I increased the maximum number of iteration to 400 LogisticRegression(solver='lbfgs', max_iter=400) and this has resolved the warning. Thank you for the explanation. — sums22, Jul 01 '20 at 09:37
Going back to this question, any tips on other things I can try to help the algorithm converge? — sums22, Aug 20 '20 at 11:54

score 54 · Answer 2 · edited May 09 '21 at 00:23

54

If you are getting the following error for any machine learning algorithm,

ConvergenceWarning:

lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

increase the number of iterations (max_iter) or scale the data as shown in 6.3. Preprocessing data

Please also refer to the documentation for alternative solver options: LogisticRegression()

Then in that case you use an algorithm like

from sklearn.linear_model import LogisticRegression
log_model = LogisticRegression(solver='lbfgs', max_iter=1000)

because sometimes it will happen due to iteration.

edited May 09 '21 at 00:23

Peter Mortensen

30,738
21
105
131

answered Mar 10 '21 at 08:25

nikhil upadhyay

882
6
2

141

This answer has been mentioned in the recent [Stack Overflow blog post](https://stackoverflow.blog/2021/04/19/how-often-do-people-actually-copy-and-paste-from-stack-overflow-now-we-know/) and, as a result, quickly gained a few upvotes today. – Sebastian Simon Apr 19 '21 at 18:11
66

Yes, and it's still a bad answer. "because sometimes it will happen due to iteration." is a fairly meaningless response... – jms Apr 20 '21 at 00:34
34

Quote from the blog post: *"most copied downvoted answer"* – Peter Mortensen Apr 21 '21 at 16:32
4

@jms still, as the post says, 'it appears to be a more concise version of the accepted answer (...) it is the perfect example of a “too long didn’t read” post.' – Guilherme Garnier Apr 28 '21 at 17:20
3

@GuilhermeGarnier sure the accepted answer is verbose, but the original poster seems to be fairly naive and probably required some extra background which the commenter was catering to. – jms Apr 28 '21 at 21:03
3

This answer has been mentioned again in [StackOverflow Blog 2021-04-19](https://stackoverflow.blog/2021/04/19/how-often-do-people-actually-copy-and-paste-from-stack-overflow-now-we-know) and will surely attract more attention :) – GoFindTruth Apr 29 '21 at 16:37
24

The blog post seems to be surprised that this answer is coped more than the accepted answer - there's nothing to copy in the accepted answer! – Phil Sep 30 '21 at 01:00
14

I came here, because it was mentioned in [how often do people actually copy and paste (2021-12-30)](https://stackoverflow.blog/2021/12/30/how-often-do-people-actually-copy-and-paste-from-stack-overflow-now-we-know/?cb=1) – Betlista Dec 30 '21 at 22:31
5

so came I, but now it **has** upvotes! – Irf Jan 02 '22 at 15:39

score 12 · Answer 3 · answered Dec 15 '21 at 23:11

12

to fix Convergence warning specify max_iter in the LogisticRegression to a higer value:

from sklearn.linear_model import LogisticRegression
model=LogisticRegression(max_iter=3000)
model.fit(X_train,y_train)

answered Dec 15 '21 at 23:11

Udesh

2,415
2
22
32

score 2 · Answer 4 · answered Jan 12 '23 at 10:08

2

What worked for me is:

from sklearn.linear_model import LogisticRegression   
logReg = LogisticRegression(solver='lbfgs', max_iter=3000)

answered Jan 12 '23 at 10:08

biihu

69
6

ConvergenceWarning: lbfgs failed to converge (status=1): STOP: TOTAL NO. of ITERATIONS REACHED LIMIT

4 Answers4

Update:

Linked