16

I am following this tutorial: http://nlp.seas.harvard.edu/2018/04/03/attention.html to implement the Transformer model from the "Attention Is All You Need" paper.

However I am getting the following error : RuntimeError: "exp" not implemented for 'torch.LongTensor'

This is the line, in the PositionalEnconding class, that is causing the error:

div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

When it is being constructed here: 

pe = PositionalEncoding(20, 0)

Any ideas?? I've already tried converting this to perhaps a Tensor Float type, but this has not worked.

I've even downloaded the whole notebook with accompanying files and the error seems to persist in the original tutorial.

Any ideas what may be causing this error?

Thanks!

noob
  • 5,954
  • 6
  • 20
  • 32
  • What is d_model in the code? – Akhilesh Pandey Oct 22 '18 at 04:53
  • @AkhileshPandey d_model is one of the parameters to initialise the PositionalEncoding class: def __init__(self, d_model, dropout, max_len=5000): Also the entire notebook can be found here: https://github.com/harvardnlp/annotated-transformer/blob/master/The%20Annotated%20Transformer.ipynb – noob Oct 22 '18 at 04:54
  • 1
    After going through the code I found that at one point the value of d_model was 20. Using this value I find that the line works fine for me – Akhilesh Pandey Oct 22 '18 at 05:02
  • Try converting it to some other type like a= a.type(torch.float36) then call torch.exp(a) – Akhilesh Pandey Oct 22 '18 at 05:11
  • I ran the code and it works fine for me. Just see if you have copied correctly. – Akhilesh Pandey Oct 22 '18 at 05:12
  • @noob Hey man, I'm trying to run this tutorial and I had this same problem. After applying the fix, I had a bunch of other similar problems, which I fixed by calling `float()` on some of the tensors and converting them to `FloatTensor`s. But now I'm getting this error: `Floating point exception: 8` and python quits. I was wondering if you had the same problem. If you didn't what did you do to fix those other problems with the float type? – user2268997 Dec 01 '18 at 19:09

5 Answers5

33

I happened to follow this tutorial too.

For me I just got the torch.arange to generate float type tensor

from 

position = torch.arange(0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

to

position = torch.arange(0., max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0., d_model, 2) * -(math.log(10000.0) / d_model))

Just a simple fix. But now it works for me. It is possible that the torch exp and sin previously support LongTensor but not anymore (not very sure about it).

LU Jialin
  • 346
  • 2
  • 4
3

It seems like torch.arange returns a LongTensor, try torch.arange(0.0, d_model, 2) to force torch to return a FloatTensor instead.

Shai
  • 111,146
  • 38
  • 238
  • 371
  • Hi, thanks for the response ! But that doesn't seem to work :( – noob Oct 22 '18 at 04:53
  • @noob are you sure this line produces this error? can you verify (in debug) that the `dtype` of `torch.arage(0, d_model, 2)` is indeed `float` and not `long`? – Shai Oct 22 '18 at 04:56
1

The suggestion given by @shai worked for me. I modified the init method of PositionalEncoding by using 0.0 in two spots:

position = torch.arange(0.0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0.0, d_model, 2) * -(math.log(10000.0) / d_model))
Joseph Cho
  • 4,033
  • 4
  • 26
  • 33
user214581
  • 11
  • 1
0

For me, installing pytorch == 1.7.1 solved the problem.

razimbres
  • 4,715
  • 5
  • 23
  • 50
0

Like Rubens said, in the higher version of Pytorch, you don't need to worry about this stuff. I can easily run it on my desktop's 1.8.0 Pytorch, but failed to go through it in my server's 1.2.0 Pytorch. There is something incompatible between different versions.

Fang WU
  • 180
  • 2
  • 7