How can I plot AUC and ROC while using fit_generator and evaluate_generator to train my network?

Question

I am using a generator to train and predict classification on my data. Here is an example of ImageDataGenerator

from keras.preprocessing.image import ImageDataGenerator

batch_size = 16

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)


train_generator = train_datagen.flow_from_directory(
        'data/train',  # this is the target directory
        target_size=(150, 150),  
        batch_size=batch_size,
        class_mode='binary') 

validation_generator = test_datagen.flow_from_directory(
        'data/validation',
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='binary')
       

model.fit_generator(
        train_generator,
        steps_per_epoch=2000 // batch_size,
        epochs=50,
        validation_data=validation_generator,
        validation_steps=800 // batch_size)
model.save_weights('first_try.h5')  # always save your weights after training or during training

My question how can I create AUC and ROC when I use fit_generator?

Answer 1

I think your best bet in this case is to define AUC as a new metric. To do this, you have to define the metric in tensorflow (I am assuming you are using tensorflow backend).

One way which I have experimented with previously (however, I don't recall I tested it for correctness of results) is something like this:

def as_keras_metric(method):
    """
    This is taken from:
    https://stackoverflow.com/questions/45947351/how-to-use-tensorflow-metrics-in-keras/50527423#50527423
    """
    @functools.wraps(method)
    def wrapper(*args, **kwargs):
        """ Wrapper for turning tensorflow metrics into keras metrics """
        value, update_op = method(*args, **kwargs)
        tf.keras.backend.get_session().run(tf.local_variables_initializer())
        with tf.control_dependencies([update_op]):
            value = tf.identity(value)
        return value
    return wrapper

and then define the metric when the model is compiled:

model.compile(metrics=['accuracy', as_keras_metric(tf.metrics.auc)], optimizer='adam', loss='categorical_crossentropy')

Although this spits out numbers, I have yet to find out if they are correct. If you are able to test this, and it gives the correct results, or not, please let me know, I would be interested to find out.

A second way to go around this is to use a callback class and define at least the on_epoch_end function, and then then you can call sklearn roc_auc_score from there and either print out or save to a log.

However, what I have found out so far, is that you need to provide it the training data through __init__, and thus with generators, you need to make sure the callback's generator is supplying the same data as the model's fitting generator. On the other hand, for the validation generator, it can be accessed from a callback class using self.validation_data, which is the same as the one supplied to fit_generator.