12

As I was going through pytorch documentation I came across a term layout = torch.strided in many of the functions. Can anyone help me in understanding where is it used and how. The description says it's the the desired layout of returned Tensor. What does layout mean and how many types of layout are there ?

torch.rand(*sizes, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False)
kmario23
  • 57,311
  • 13
  • 161
  • 150
Bhanujeet Choudhary
  • 301
  • 1
  • 2
  • 10

4 Answers4

13

strides is number of steps (or jumps) that is needed to go from one element to next element, in a given dimension. In computer memory, the data is stored linearly in a contiguous block of memory. What we view is just a (re)presentation.

Let's take an example tensor for understanding this:

# a 2D tensor
In [62]: tensor = torch.arange(1, 16).reshape(3, 5) 

In [63]: tensor  
Out[63]: 
tensor([[ 1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10],
        [11, 12, 13, 14, 15]])

With this tensor in place, the strides are:

# get the strides
In [64]: tensor.stride() 
Out[64]: (5, 1)

What this resultant tuple (5, 1) says is:

  • to traverse along the 0th dimension/axis (Y-axis), let's say we want to jump from 1 to 6, we should take 5 steps (or jumps)
  • to traverse along the 1st dimension/axis (X-axis), let's say we want to jump from 7 to 8, we should take 1 step (or jump)

The order (or index) of 5 & 1 in the tuple represents the dimension/axis. You can also pass the dimension, for which you want the stride, as an argument:

# get stride for axis 0
In [65]: tensor.stride(0) 
Out[65]: 5

# get stride for axis 1
In [66]: tensor.stride(1) 
Out[66]: 1

With that understanding, we might have to ask why is this extra parameter needed when we create the tensors? The answer to that is for efficiency reasons. (How can we store/read/access the elements in the (sparse) tensor most efficiently?).

With sparse tensors (a tensor where most of the elements are just zeroes), so we don't want to store these values. we only store the non-zero values and their indices. With a desired shape, the rest of the values can then be filled with zeroes, yielding the desired sparse tensor.

For further reading on this, the following articles might be of help:

P.S: I guess there's a typo in the torch.layout documentation which says

Strides are a list of integers ...

The composite data type returned by tensor.stride() is a tuple, not a list.

kmario23
  • 57,311
  • 13
  • 161
  • 150
1

For quick understanding, layout=torch.strided corresponds to dense tensors while layout=torch.sparse_coo corresponds to sparse tensors.

From another perspective, we can understand it together with torch.tensor.view. A tensor can be viewed indicates it is contiguous. If we change the view of a tensor, the strides will change accordingly, but the data will keep the same. More specifically, view returns a new tensor with the same data but different shape, and strides is compatible with the view to indicate how to access the data in the memory.

For example

In [1]: import torch
In [2]: a = torch.arange(15)

In [3]: a.data_ptr()
Out[3]: 94270437164688

In [4]: a.stride()
Out[4]: (1,)

In [5]: a = a.view(3, 5)

In [6]: a.data_ptr() # share the same data pointer
Out[6]: 94270437164688

In [7]: a.stride() # the stride changes as the view changes
Out[7]: (5, 1)

In addition, the idea of torch.strided is basically the same as strides in numpy. View this question for more detailed understanding. How to understand numpy strides for layman?

drowsyleilei
  • 438
  • 4
  • 9
0

As per the official pytorch documentation here,

A torch.layout is an object that represents the memory layout of a torch.Tensor. Currently, we support torch.strided (dense Tensors) and have experimental support for torch.sparse_coo (sparse COO Tensors).

torch.strided represents dense Tensors and is the memory layout that is most commonly used. Each strided tensor has an associated torch.Storage, which holds its data. These tensors provide multi-dimensional, strided view of a storage. Strides are a list of integers: the k-th stride represents the jump in the memory necessary to go from one element to the next one in the k-th dimension of the Tensor. This concept makes it possible to perform many tensor operations efficiently.

Example:

>>> x = torch.Tensor([[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]])
>>> x.stride()
(5, 1)

>>> x.t().stride()
(1, 5)
Anubhav Singh
  • 8,321
  • 4
  • 25
  • 43
0

layout means the way memory organize the element in that tensor,I think currently there are 2 types of layout to store tensor one is torch.strided and another is torch.sparse_coo

strided means the element is arranged one by one in a very dense way, think about a strided troops, squares,so each soldier actually has neighbours.

while for sparse_coo I think should be deal with sparse matrix, the exact storage structure I am not sure but I guess it just stores non-zero elements' indices and values

It need to separate for those two types because for sparse matrix no need to arrange element one by one in a dense form, because it will take maybe one hundred steps for the non-zero element get to its next non-zero elements

cloudscomputes
  • 1,278
  • 13
  • 19