0

I am building the model given in this paper.

I came across a function to reshape one of the input embeddings, however I am not sure how it works.

values = values.reshape(N, value_len, self.heads, self.head_dim)
#Here N, value_len and embed size is the dimension of the tensor "values"

I have tried the official documentation but could not make sense of what this function is exactly doing?

Anwesh saha
  • 11
  • 3

1 Answers1

0

When the tensor is contiguous, the reshape function does not modify the underlying tensor data. It only returns a different view on that tensor's data such that it gets the proper form to be called on other functions. Otherwise, if the tensor is non-contiguous, it will return a copy of that tensor. In this case, the values themselves remain the same but the layout changes.

Views on tensors are used to make sense of what the object contains, and how it should be regarded as. For example, a tensor with a shape of (H, W) would be an image, while a shape of (B, D, C, H, W) might correspond to a batch of point clouds, etc...

Ivan
  • 34,531
  • 8
  • 55
  • 100
  • So in the example above, what exactly is happening to the tensor "values"? – Anwesh saha Mar 09 '23 at 11:53
  • It changes the view of the tensor to the shape `(N, value_len, self.heads, self.head_dim)`. – Ivan Mar 09 '23 at 13:00
  • 1
    This is not quite entirely right. If the original tensor is contiguous, then `reshape` returns the same thing as `view` (i.e. a tensor view) but if the original tensor is not contiguous, `reshape` creates a new, contiguous copy of the data with the desired shape. See: https://stackoverflow.com/questions/69840389/what-functions-or-modules-require-contiguous-input/69854867#69854867 – DerekG Mar 09 '23 at 14:46
  • 1
    Yes this is very true @DerekG, the layout of the data can change however the values themselves don't, that's what I wanted to point out but it wasn't made clear in my response. I have edited my answer. – Ivan Mar 09 '23 at 15:40