In Tensorflow, adding data augmentation layers to my keras model slows down training by over 10x

Question

I'm adding data augmentation to my tensorflow model like so:

data_augmentation = keras.Sequential([
  layers.experimental.preprocessing.RandomRotation(factor=0.4, fill_mode="wrap"),
  layers.experimental.preprocessing.RandomTranslation(height_factor=0.2, width_factor=0.2, fill_mode="wrap"),
  layers.experimental.preprocessing.RandomFlip("horizontal"),
  layers.experimental.preprocessing.RandomContrast(factor=0.2),
  layers.experimental.preprocessing.RandomHeight(factor=0.2),
  layers.experimental.preprocessing.RandomWidth(factor=0.2)
])

input_shape = (299, 299, 3)

inceptionV3_base = tf.keras.applications.InceptionV3(
    input_shape=input_shape,
    include_top=False,
    weights='imagenet'
)

tf_model = tf.keras.Sequential([
    tf.keras.layers.InputLayer(input_shape=input_shape),
    data_augmentation,
    inceptionV3_base,
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dense(1024, activation='relu'),
    tf.keras.layers.Dense(len(classes), activation='softmax')
])

Adding the data_augmentation layer to model slows down training by 13x. Am I using the keras preprocessing layers correctly?

To get more insight, you could use the [tensorflow profiler](https://www.tensorflow.org/guide/profiler) — Lescurel, Jan 07 '21 at 09:13

score 0 · Answer 1 · answered May 13 '21 at 14:15

0

I had the same issue - was missing ptxas (nvidia-cuda-toolkit package for me).

answered May 13 '21 at 14:15

xl0

1

score 0 · Answer 2 · answered Aug 30 '22 at 23:16

There are two ways of adding data augmentation: 1- Inside the model, just like the way you did. 2- Outside the model, and before training, using tf.data.Dataset.map()

Maybe trying option2 could make your model training faster. Try it! More details here: https://keras.io/guides/preprocessing_layers/

There are many other ways of optimizing the training, such as the way you feed data into your model. Are you using tf.data.Dataset methods like cache() and prefetch(). More details can be found here: https://www.tensorflow.org/tutorials/load_data/images#configure_the_dataset_for_performance

In Tensorflow, adding data augmentation layers to my keras model slows down training by over 10x

2 Answers2