3

if I tune a model with the LightGBMTunerCV I always get this massive result of the cv_agg's binary_logloss. If I do this with a bigger dataset, this (unnecessary) io slows down the performance of the optimization process.

Here is the code:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import optuna.integration.lightgbm as lgb
import optuna

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=UserWarning)
breast_cancer = load_breast_cancer()

X_train, X_test, Y_train, Y_test = train_test_split(breast_cancer.data, breast_cancer.target)

train_dataset = lgb.Dataset(X_train, Y_train, feature_name=breast_cancer.feature_names.tolist())
test_dataset = lgb.Dataset(X_test, Y_test, feature_name=breast_cancer.feature_names.tolist())
callbacks = [lgb.log_evaluation(period=0)]
tuner = lgb.LightGBMTunerCV({"objective": "binary", 'verbose': -1},
       train_set=test_dataset, num_boost_round=10,
       nfold=5, stratified=True, shuffle=True)


tuner.run()

And the output:

feature_fraction, val_score: 0.327411:  43%|###################2      | 3/7 [00:00<00:00, 13.84it/s]
[1] cv_agg's binary_logloss: 0.609496 + 0.009315
[2] cv_agg's binary_logloss: 0.554522 + 0.00607596
[3] cv_agg's binary_logloss: 0.512217 + 0.0132959
[4] cv_agg's binary_logloss: 0.479142 + 0.0168108
[5] cv_agg's binary_logloss: 0.440044 + 0.0166129
[6] cv_agg's binary_logloss: 0.40653 + 0.0200005
[7] cv_agg's binary_logloss: 0.382273 + 0.0242429
[8] cv_agg's binary_logloss: 0.363559 + 0.03312

Is there any way to get rid of this output?

Thanks for the help!

NonStopAggroPop
  • 55
  • 1
  • 6
  • What happens if you call set_verbosity beforehand to CRITICAL only? https://optuna.readthedocs.io/en/stable/reference/generated/optuna.logging.set_verbosity.html#optuna.logging.set_verbosity (the verbosity parameter to LightGBMTunerCV is deprecated in favour of this.) – SamBob Nov 24 '21 at 09:01
  • Unfortunately, setting ```optuna.logging.set_verbosity(optuna.logging.CRITICAL)``` does not mute any more lines. – NonStopAggroPop Nov 24 '21 at 09:09
  • 1
    Shame. What about redirecting stdout to null with `contextlib` for that call https://stackoverflow.com/a/37989355/1581658 ? It might save some performance? – SamBob Nov 24 '21 at 09:17
  • That works! Although I think this is more of a workaround, since I am piping the stdout to a logfile, so I would be missing some warnings that might be thrown. I think there should be a more direct parameter for that. But I could not find anything in the documentation. So your suggestion is very much appreciated! Thanks! – NonStopAggroPop Nov 24 '21 at 09:31

1 Answers1

3

You can pass verbose_eval parameter with value None in LightGBMTunerCV().

Example:

tuner = lgb.LightGBMTunerCV({"objective": "binary", 'verbose': -1},
       train_set=test_dataset, num_boost_round=10,
       nfold=5, stratified=True, shuffle=True, verbose_eval=None)
ferdy
  • 4,396
  • 2
  • 4
  • 16
  • This is exactly what I was looking for. Setting verbose_eval to None was not my expected input for muting the output. Thank you! – NonStopAggroPop Nov 30 '21 at 09:40