I'm trying to train a TensorFlow (version 2.11.0) code in float16.
I checked that FP16 is supported on the RTX 3090 GPU. So, I followed the below link to train the whole code in reduced precision.
https://www.tensorflow.org/api_docs/python/tf/keras/backend/set_floatx
Then, When I was testing the training time of the model, I observed that the training time is not reduced. Interestingly when I manually convert some precision try to change the precision for some parts of code, the training time gets increased (due to the overhead of converting to FP16).
According to the above link it was mentioned that by default the code will run in float16, but is there any option to check if the underlying hardware really runs in float16?
Furthermore, my network is a 3-layer MLP network (which is quite small), is there any change that FP16 works only with larger networks?
Note:
I'm trying to train an Reinforcement Learning algorithm in reduced precision
(link: https://github.com/openai/maddpg)
I used CUDA 9.0, cuDNN 7.6.5 to train the algorithm
