1

I am studying the transformer code here as an example.

The positional encoding function below uses numpy for all of its operations. Then it casts them back to TF tensor when returning the result. Does such a pattern impact the performance when running the code especially when I run it on an GPU? Is that recommended to use only TF operations when implementing a model? 

def get_positional_encoding(self, max_len):
    """PE_(pos, 2i) = sin(pos/10000^(2i/d_model))
    PE_(pos, 2i+1) = cos(pos/10000^(2i/d_model))
    """

    pos = np.expand_dims(np.arange(0, max_len), axis=1)
    div_term = np.array([[1 / np.power(10000, (2 * (i//2) / self.d_model)) for i in range(self.d_model)]])       

    pos = pos * div_term

    pe = np.zeros((max_len, self.d_model))
    pe[:, 0:self.d_model//2] = np.sin(pos[:, 0::2])
    pe[:, self.d_model//2:] = np.cos(pos[:, 0::2])

    pe = np.expand_dims(pe, 0)

    print(pe.shape)

    return tf.cast(pe, dtype=tf.float32)
Ruolin
  • 145
  • 1
  • 1
  • 7
  • I think tensor board will help you in this,check if the GPU performance in both the cases,check out [this](https://stackoverflow.com/questions/45544603/tensorflow-how-do-you-monitor-gpu-performance-during-model-training-in-real-tim) – Shubham Shaswat Feb 17 '20 at 13:07
  • you cannot use numpy in your model implementation, because tensorflow cannot differentiate through numpy operations. in other words, having numpy operations in a tensorflow model would break the gradient flow and the model would not be train-able. – jkr Feb 17 '20 at 14:12

1 Answers1

1

Yes, it does impact the performance generally this is assuming that you can properly execute the code without any errors.

One factor is the time required passing/copying the values from CPU to GPU, which adds a lot of overhead time, the larger the matrices.

Tensorflow is built to run in the GPU, so when you are using all Tensorflow operations in every part of your code, you will see a drastic improvement in performance just because of the time required in passing values from CPU to GPU, this is not considering the optimizations applied when using Tensors for computations.

TF_Support
  • 1,794
  • 1
  • 7
  • 16