22

I'm currently working on building a convolutional neural network (CNN) that will work on financial time series data. The input shape is (100, 40) - 100 time stamps by 40 features.

The CNN that I'm using uses asymmetric kernel sizes (i.e. 1 x 2 and 4 x 1) and also asymmetric strides (i.e. 1 x 2 for the 1 x 2 layers and 1 x 1 for the 4 x 1 layers).

In order to maintain the height dimension to stay 100, I needed to pad the data. In my research, I noticed that people who use TensorFlow or Keras simply use padding='same'; but this option is apparently unavailable in PyTorch.

According to some answers in What is the difference between 'SAME' and 'VALID' padding in tf.nn.max_pool of tensorflow?, and also this answer on the PyTorch discussion forum, I can manually calculate how I need to pad my data, and use torch.nn.ZeroPad2d to solve the problem - since apparently normal torch.nn.Conv2d layers don't support asymmetric padding (I believe that the total padding I need is 3 in height and 0 in width).

I tried this code:

import torch
import torch.nn as nn

conv = nn.Conv2d(1, 1, kernel_size=(4, 1))
pad = nn.ZeroPad2d((0, 0, 2, 1)) # Add 2 to top and 1 to bottom.

x = torch.randint(low=0, high=9, size=(100, 40))
x = x.unsqueeze(0).unsqueeze(0)

y = pad(x)

x.shape # (1, 1, 100, 40)
y.shape # (1, 1, 103, 40)

print(conv(x.float()).shape)
print(conv(y.float()).shape)

# Output
# x -> (1, 1, 97, 40)
# y -> (1, 1, 100, 40)

It does work, in the sense that the data shape remains the same. However, is there really no padding='same' option available? Also, how can we decide which side to pad?

Karl Knechtel
  • 62,466
  • 11
  • 102
  • 153
Sean
  • 2,890
  • 8
  • 36
  • 78
  • https://stackoverflow.com/questions/55140554/convolutional-encoder-error-runtimeerror-input-and-target-shapes-do-not-matc/55143487#55143487 have a look at that, it will give you a clue. – Salih Karagoz Oct 09 '19 at 15:05
  • 1
    Really speaking, that feature is possible in Tensorflow due to its static computation graph. In PyTorch, there is a dynamic computation graph, so it's probably difficult to implement (otherwise they would have already done that). Within `nn.Conv2D`, as you say, there is only symmetric padding, but different padding can be done along different dimensions. – akshayk07 Oct 09 '19 at 15:08
  • 1
    I think @akshayk07 is right and the dynamic nature of pytorch makes it hard; Here is a good implementation of 'same' padding in pytorch (for 2d conv): https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/padding.py#L28 – Separius Apr 09 '20 at 06:13

3 Answers3

8

I had the same issue some time ago, so I implemented it myself using a ZeroPad2d layer as you are trying to do. Here is the right formula:

from functools import reduce
from operator import __add__

kernel_sizes = (4, 1)

# Internal parameters used to reproduce Tensorflow "Same" padding.
# For some reasons, padding dimensions are reversed wrt kernel sizes,
# first comes width then height in the 2D case.
conv_padding = reduce(__add__, 
    [(k // 2 + (k - 2 * (k // 2)) - 1, k // 2) for k in kernel_sizes[::-1]])

pad = nn.ZeroPad2d(conv_padding)
conv = nn.Conv2d(1, 1, kernel_size=kernel_sizes)

print(x.shape) # (1, 1, 103, 40)
print(conv(y.float()).shape) # (1, 1, 103, 40)

Also, as mentioned by @akshayk07 and @Separius, I can confirm that it is the dynamic nature of pytorch that makes it hard. Here is a post about this point from a Pytorch developper.

milembar
  • 919
  • 13
  • 17
8

It looks like there is now, in pytorch 1.9.1, according to the docs.

padding='valid' is the same as no padding. padding='same' pads the input so the output has the shape as the input. However, this mode doesn't support any stride values other than 1.

lucidbrot
  • 5,378
  • 3
  • 39
  • 68
  • what value does padding='same' use? 0? same as last pixel? miror? – Ofir Shifman Jun 06 '23 at 13:56
  • 1
    @OfirShifman see also [in the docs](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html): `padding_mode (str, optional) – 'zeros', 'reflect', 'replicate' or 'circular'. Default: 'zeros'`. That means you can choose but by default it will pad with zeroes. – lucidbrot Jun 06 '23 at 14:39
1

padding='same' and padding='valid' is possible in Pytorch 1.10.0+. However, 'same' and 'valid' for padding is not possible for when stride > 1.

rocksyne
  • 1,264
  • 15
  • 17