A worker process managed by the executor was unexpectedly terminated

Question

Hello I am executing the following code:

grid_params_bc = {
        'base_estimator': [best_knn, best_rf, best_l],
        'n_estimators': [10, 20, 50]
    }

    gs_bc = GridSearchCV(
        BaggingClassifier(),
        grid_params_bc,
        cv=10,
        verbose=1,
        n_jobs=-1,
        return_train_score=True,
        scoring='f1_micro'
    )

    clf_bc = gs_bc.fit(X_train, y_train)

But I am getting the following error:

A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

Anyway I could execute the first 34 task of 80.

Why this error is happing?

Thanks

Will this help? https://stackoverflow.com/a/54141605/13552470 — Red, Jun 25 '20 at 13:00

score 1 · Accepted Answer · answered Jun 25 '20 at 13:14

Without knowing the specifics of your data or the actual classifiers you are using, the best guess is that you are doing too heavy computations on your system. As the error message says, there might be an excessive use of memory.

You are performing a grid search on a BaggingClassifier with up to 50 base estimators such as Random Forests (at least that's how it looks like). Since each Random Forest consists of 100 Decision Trees by default, this would e.g. result in training and testing 5000 Decision Trees, just in that instance.

Try to use lower numbers of base estimators n_estimators and maybe change the n_jobs parameter. You have to try around, but it should resolve the issue.

A worker process managed by the executor was unexpectedly terminated

1 Answers1