After downloading an EfficientNet model from tensorflow.keras.applications.efficientnet, and retraining it on our own data, I've noticed that the results are not reproducible. The results are reproducible for other architectures like VGG16, ResNet101, InceptionV3, and InceptionResNetV2, but not for any of the EfficientNetBx models.
Layer by layer analysis shows that the DepthwiseConv2D layer is producing different gradients. I also tried the MobileNetV2 which has the same type of layers, and it is NOT reproducible either. I wonder if anyone else has encountered this issue and how they have solved it.
Please note that I've set all the following seeds, and even have tensorflow-determinism:
random.seed(1)
np.random.seed(1)
tf.random.set_seed(1)
os.environ['TF_CUDNN_DETERMINISTIC'] = TRUE
os.environ['TF_DETERMINISTIC_OPS'] = TRUE
TensorFlow Version: tensorflow-gpu==2.3
Opened this issue on Tensorflow GitHub: https://github.com/tensorflow/tensorflow/issues/47174
