5

Is there any way we can target a specific metric to optimize using inbuilt tensorflow optimizers? If not, how to achieve this? For eg. If I want to focus only on maximizing F-score of my classifier specifically, is it possible to do so in tensorflow?

estimator = tf.estimator.LinearClassifier(
  feature_columns=feature_cols,
  config=my_checkpointing_config,
  model_dir=output_dir,
  optimizer=lambda: tf.train.FtrlOptimizer(
      learning_rate=tf.train.exponential_decay(
          learning_rate=0.1,
          global_step=tf.train.get_or_create_global_step(),
          decay_steps=1000,
          decay_rate=0.96)))

I am trying to optimize my classifier specifically on the basis of getting a better F-score. Despite using the decaying learning_rate and 300 training steps I am getting inconsistent results. While checking the metrics in the logs, I found the behavior of precision, recall and accuracy to be very erratic. Despite increasing the number of training steps, there was no significant improvement. So I thought that if i could make the optimizer focus more on improving the F-score as a whole I might get better results. Hence the question. Is there something that I am missing?

Pulah
  • 123
  • 12

2 Answers2

5

In classification settings, optimizers minimize the loss, e.g. cross entropy; quantities like accuracy, F-score, precision, recall etc. are essentially business metrics, and they are not (and cannot be) directly minimized during the optimization process.

This is a question that pops up rather frequently here in SO in various disguises; here are some threads which will hopefully help you disentangle the concepts (although they refer to accuracy, precision, and recall, the argument is exactly the same for the F-score):

Loss & accuracy - Are these reasonable learning curves?

Cost function training target versus accuracy desired goal

Is there an optimizer in keras based on precision or recall instead of loss?

The bottom line, adapting one of my own (linked) answers:

Loss and metrics like accuracy or F-score are different things; roughly speaking, metrics like accuracy & F-score are what we are actually interested in from a business perspective, while the loss is the objective function that the learning algorithms (optimizers) are trying to minimize from a mathematical perspective. Even more roughly speaking, you can think of the loss as the "translation" of the business objective (accuracy, F-score etc) to the mathematical domain, a translation which is necessary in classification problems (in regression ones, usually the loss and the business objective are the same, or at least can be the same in principle, e.g. the RMSE)...

desertnaut
  • 57,590
  • 26
  • 140
  • 166
0

One could technically adjust the threshold parameter that distinguishes between class 1 and 0. For example, in logistic regression, if the threshold is lowered from 0.5 to 0.3, recall would decrease and precision would increase, and viceversa. But as others have mentioned, this is not the same as optimizing ("minimizing") the loss function.

Odisseo
  • 747
  • 1
  • 13
  • 32