Upweight a Category

Question

I have built a TensorFlow model that uses a DNNClassifier to classify input into two categories.

My problem is that Outcome 1 occurs upwards of 90-95% of the time. Therefore, TensorFlow is giving me the same probabilities for all of my predictions.

I am trying to predict the other outcome (e.g. having a false positive for Outcome 2 is preferable to missing a possible occurrence of Outcome 2). I know that in machine learning in general, in this case it would be worthwhile to try to upweight Outcome 2.

However, I don't know how to do this in TensorFlow. The documentation alludes to it being possible, but I can't find any examples of what it would actually look like. Has anyone has successfully done this, or does anyone know where I could find some example code or a thorough explanation (I'm using Python)?

Note: I have seen exposed weights being manipulated when someone is using the more fundamental parts of TensorFlow and not an estimator. For maintenance reasons, I need to do this using an estimator.

here an updated example: https://stackoverflow.com/questions/52383967/imbalanced-classes-in-multi-class-classification-problem/62385978#62385978 — Marco Cerliani, Jun 15 '20 at 10:17

Maxim · Accepted Answer · 2018-01-05T14:07:02.400

8

tf.estimator.DNNClassifier constructor has weight_column argument:

weight_column: A string or a _NumericColumn created by tf.feature_column.numeric_column defining feature column representing weights. It is used to down weight or boost examples during training. It will be multiplied by the loss of the example. If it is a string, it is used as a key to fetch weight tensor from the features. If it is a _NumericColumn, raw tensor is fetched by key weight_column.key, then weight_column.normalizer_fn is applied on it to get weight tensor.

So just add a new column and fill it with some weight for the rare class:

weight = tf.feature_column.numeric_column('weight')
...
tf.estimator.DNNClassifier(..., weight_column=weight)

[Update] Here's a complete working example:

import numpy as np
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets('mnist', one_hot=False)
train_x, train_y = mnist.train.next_batch(1024)
test_x, test_y = mnist.test.images, mnist.test.labels

x_column = tf.feature_column.numeric_column('x', shape=[784])
weight_column = tf.feature_column.numeric_column('weight')
classifier = tf.estimator.DNNClassifier(feature_columns=[x_column],
                                        hidden_units=[100, 100],
                                        weight_column=weight_column,
                                        n_classes=10)

# Training
train_input_fn = tf.estimator.inputs.numpy_input_fn(x={'x': train_x, 'weight': np.ones(train_x.shape[0])},
                                                    y=train_y.astype(np.int32),
                                                    num_epochs=None, shuffle=True)
classifier.train(input_fn=train_input_fn, steps=1000)

# Testing
test_input_fn = tf.estimator.inputs.numpy_input_fn(x={'x': test_x, 'weight': np.ones(test_x.shape[0])},
                                                   y=test_y.astype(np.int32),
                                                   num_epochs=1, shuffle=False)
acc = classifier.evaluate(input_fn=test_input_fn)
print('Test Accuracy: %.3f' % acc['accuracy'])

edited Jan 05 '18 at 14:07

answered Jan 04 '18 at 16:16

Maxim

52,561
27
155
209

1

I did see this, but the confusing part is that it appears to be weighting individual pieces of data, instead of weighting by class. I'm sure there's a workaround where I could create a numeric column with the correct weight for each row by scanning through the data set first, but it seems like a cumbersome way to do it. – Abigail Fox Jan 04 '18 at 16:21
@AbigailFox You're right, but it is so. See [this answer](https://stackoverflow.com/a/47034889/712995) and specifically the part about weights. `DNNClassifier` uses `tf.nn.sigmoid_cross_entropy_with_logits` loss for binary classification. – Maxim Jan 04 '18 at 16:26
I created a new column with the proper weights, but it's giving me a `Value Error` with `not in feature dictionary` if I only use the weight column within the declaration of the `DNNClassifier`. That error disappears if I use the `weights` column as a feature column in the model, but I believe that would skew my results/not give the correct output. – Abigail Fox Jan 04 '18 at 16:46
What is your tensorflow version? Works for me in 1.4.1. I also don't see this check in the source code of tensorflow. – Maxim Jan 04 '18 at 16:52
@AbigailFox See an update: I've added a complete example. I think you forgot to return `weight` from the input function and that caused an error. – Maxim Jan 05 '18 at 14:07
Thank you! I'm at least getting different outputs for each one now, I'm going to keep playing around with it in hopes of improving accuracy. – Abigail Fox Jan 05 '18 at 14:41
Okay I have a big concern about this, and that is feeding weights into the input function. Is simply identifying which column is used to weight when declaring the DNNClassifier enough to make sure that the model isn't training on the weights (which would essentially ruin the model)? – Abigail Fox Jan 08 '18 at 14:33
If it's not in feature columns, it shouldn't be seen by the layers. You can inspect the shape of your weights to make sure of that, e.g., in tensorboard or manually – Maxim Jan 08 '18 at 14:37
That is giving me the "weights are not in feature dictionary" error though. I don't see this well-described in the documentation either. – Abigail Fox Jan 08 '18 at 14:42
You've written the same several days ago. Look at my example: `weight_column` is not in `feature_columns` – Maxim Jan 08 '18 at 14:47
I figured it out, thank you! My input_fn was just structured a bit differently, where was where I got confused. – Abigail Fox Jan 10 '18 at 13:20
Should the `feature_columns` be initialized with some values before training? – ezy Jun 15 '20 at 07:24

Upweight a Category

1 Answers1

Linked