23

Is there a way to pad a tensor of variable size to a given shape with a specific pad value? For example given the tensors:

[[1, 2],
 [3, 4]]

and 

[[1, 2, 3],
 [4, 5, 6]]

Is there a way to have a generic operation which would take either and pad them with a value (say, to shape [2, 4] with value -1) to result in:

[[1, 2, -1, -1],
 [3, 4, -1, -1]]

and

[[1, 2, 3, -1],
 [4, 5, 6, -1]]

respectively? My reasoning (in case there is a better solution) is that I have examples from a TFRecords file, part of which has a variable length. For processing, a static length makes them easier to work with.

golmschenk
  • 11,736
  • 20
  • 78
  • 137

4 Answers4

32

Yes. There is. Provided you do not need to change the rank of the tensor, it's very simple.

tf.pad() accepts regular python lists with tensors. The format of the padding is a list of pairs of how much to pad on each side of that dimension.

e.g.

t = tf.constant([[1, 2], [3, 4]])
paddings = [[0, 0], [0, 4-tf.shape(t)[0]]]
out = tf.pad(t, paddings, 'CONSTANT', constant_values=-1)
sess.run(out)
# gives: 
# array([[ 1,  2, -1, -1],
#       [ 3,  4, -1, -1]], dtype=int32)

If you want to generalise this to a useful function, you could do something like:

def pad_up_to(t, max_in_dims, constant_values):
    diff = max_in_dims - tf.shape(t)
    paddings = tf.pad(diff[:, None], [[0, 0], [1, 0]])
    return tf.pad(t, paddings, 'CONSTANT', constant_values=constant_values)
# (note: see edits for the solution referred to by other answers on this question)

where max_in_dims is essentially the desired shape of the output. Note: this function will fail if you provide a shape that is strictly smaller than t in any dimension.

You can use it like:

t = tf.constant([[1, 2], [3, 4]]) # shape = [2, 2]
t_padded = pad_up_to(t, [2, 4], -1) # shape = [2, 4], padded with -1s

or

t = tf.placeholder(tf.float32, [None, None]) # shape = [?, ?]
t_padded = pad_up_to(t, [5,5], -1) # shape = [5, 5], padded with -1s
t_np = np.random.uniform(0, 1, [3,4]) # shape = [3,4], no padding
t_padded_out = sess.run(t_padded, {t: t_np})
t_np2 = np.random.uniform(0, 1, [2,1]) # shape = [2,1], no padding
t_padded_out2 = sess.run(t_padded, {t: t_np2})

Although the dimension sizes are calculated dynamically, the number of dimensions is not, so make sure that max_in_dims has the same number of elements as t.shape.

Multihunter
  • 5,520
  • 2
  • 25
  • 38
  • 3
    What if t has a dynamic size (e.g., its size is determined only after some placeholder is fed)? – TNg Jul 10 '18 at 04:37
  • In my provided function, `s` is a tensor that is the shape of `t`, so the amount to pad is calculated dynamically. The number of dimensions is not calculated dynamically, so just make sure your `max_in_dims` is a vector with has the same number of elements as your `t` has dimensions. If you do this it will just work (I wrote the function with this use-case in mind). – Multihunter Jul 10 '18 at 04:57
  • I didn't expect it to work with a dynamic size but to my surprise, it does! Thanks! – TNg Jul 10 '18 at 07:50
  • Good reference to not waste time finding a more off the shelf solution. – John Jiang Nov 20 '19 at 17:49
  • 1
    This didn't really work for me in TF 2.3 with dynamic sizes since `m` is evaluated to `None` which throws an error for the subtraction. However, the fix is to simply change the line to `[[0, m - s[i]] if m != None else [0,0] for (i, m) in enumerate(max_in_dims)]`. – runDOSrun Sep 04 '20 at 08:32
  • enumerate doesnt work in Graph Mode im afraid.. – litterbugkid Feb 23 '23 at 10:49
  • @Neeta Seems to work fine for me https://gist.github.com/Multihuntr/c744b1ae7a05bcb5611b273e1160152a (Tested on Tensorflow 2.11; latest docker image for tensorflow; I'm as surprised as you that this function still works 5 years later) – Multihunter Feb 27 '23 at 03:57
  • @Multihunter have you tried saving a tensorflow inference model where you're using enumerate somewhere? – litterbugkid Mar 10 '23 at 14:40
  • @Neeta Nope! Got a code snippet to show? – Multihunter Mar 14 '23 at 02:57
  • @Multihunter havent had a chance to get this working for above question but this worked for me and is compatible when saving a model which creates an execution Graph as part of that process.. `paddings = tf.reshape(tf.concat([tf.constant([0]),tf.subtract(tf.shape(products),tf.shape(scores))],0),[1,2]) scores = tf.pad(scores, paddings, 'CONSTANT', constant_values=np.nan)` – litterbugkid Mar 15 '23 at 14:39
  • I was actually asking for a code snippet to show the solution not working. :sweat_smile: Regardless, I've updated the answer with a version that should work for any exporting (based on your code snippet). – Multihunter Mar 16 '23 at 01:10
  • @Multihunter your original code was what didn't work. Cool thanks. – litterbugkid Mar 21 '23 at 12:37
3

An extension of Multihunter's solution so that padding is only performed when necessary and does not yield an error for longer inputs:

Suppose we have a sequential input called inp_seq, which is a tensor of rank 4 and should be padded in order to have a minimum length of filter_size in dimension 1.

def dynamic_padding(inp, min_size):

    pad_size = min_size - tf.shape(inp)[1]
    paddings = [[0, 0], [0, pad_size], [0, 0], [0, 0]] # assign here, during graph execution
    return tf.pad(inp, paddings)

# Pad only if necessary
padded = tf.cond(tf.less(tf.shape(inp_seq)[1], filter_size), true_fn=lambda: dynamic_padding(inp_seq, filter_size), false_fn=lambda: inp_seq)
Ataxias
  • 1,085
  • 13
  • 23
  • The line creating a tf.Variable is redundant, since the subsequent line overwrites it with a python list. You can remove that line and it will function the same. (Also, a `sequence` is a class defined by the python base libraries, while a `tensor` is defined by tensorflow: I think you should clarify which of these your `inp_seq` actually is; I presume that what you're dealing with is actually a sequence (or list) of `Tensors` like `inp_seq=[Tensor, Tensor, Tensor]`) – Multihunter Sep 03 '18 at 04:54
  • I removed the redundant line, thank you for the suggestion. The input is simply a tensor; I used the term sequence with its broader meaning (to refer to data of high dimensionality which are sequential along one dimension, namely the one to pad), I was not referring to the python base libraries. I clarified this in the edit. – Ataxias Sep 04 '18 at 19:14
1

I ran into something similar. Not fully general, but you can do something like:

test_a = tf.constant([[1, 2], [3, 4]])
test_b = tf.constant([[1, 2, 3], [4, 5, 6]])

def pad_second_dim(input, desired_size):
    padding = tf.tile([[0]], tf.stack([tf.shape(input)[0], desired_size - tf.shape(input)[1]], 0))
    return tf.concat([input, padding], 1)

with tf.Session() as sess:
    print sess.run(pad_second_dim(test_a, 4))
    # [[1 2 0 0] [3 4 0 0]]
    print sess.run(pad_second_dim(test_b, 4))
    # [[1 2 3 0] [4 5 6 0]]
WuTheFWasThat
  • 583
  • 1
  • 5
  • 11
0

The accepted answer didn't work for me either and I am reluctant to do assignments in the graph. Here, I adjusted Multihunter's answer in such a way that it should work with variable sizes. A variation on this worked for me. Specifically, I am consuming data with tf.data.TFREcordDataset and wanted to apply padding on load instead of writing out the data pre-padded.

MIN_SIZE = 100

# v shape is undefined on the second dimension. 
v = tf.get_variable(shape=(2, None), dtype=tf.int32)

# Note: this will blow up if `padding_len` < 0
padding_len = MIN_SIZE - tf.shape(v)[-1]

# paddings = [ [0, 0], [0, padding_len] ]
paddings = tf.concat( ([[0, 0]], [[0, padding_len ]]), axis=0)

# padded.shape = (2, 100)
padded = tf.pad(t, paddings, 'CONSTANT', constant_values=-1)
Russell
  • 1,252
  • 15
  • 21
  • I think what you are doing essentially is defining a tensor `paddings`, instead of a python list, to be used as an argument in `tf.pad`. In my original answer I was doing the same thing, as I would get errors otherwise. It turns out that for newer Tensorflow versions (at least 1.10), both Multihunter's solution and my newest one work. – Ataxias Sep 06 '18 at 01:26