Dynamic size for tf.zeros() (for use with placeholders with None dimensions)

Question

Consider the following code:

x = tf.placeholder("float", shape=[42, 4])
y = tf.zeros([42, 4], "float")
xy_stacked = tf.concat(1, [x, y])

print(x.get_shape())
print(y.get_shape())
print(xy_stacked.get_shape())

This will produce the following output, as expected:

TensorShape([Dimension(42), Dimension(4)])
TensorShape([Dimension(42), Dimension(4)])
TensorShape([Dimension(42), Dimension(8)])

However, what if the placeholder has a dynamic dimension that is determined at run-time by the value passed to feed_dict=, as placeholders often do:

x = tf.placeholder("float", shape=[None, 4])
y = tf.zeros([None, 4], "float")
xy_stacked = tf.concat(1, [x, y])

This will produce an error for tf.zeros([None, 4], "float"). Apparently Dimension(None) is not allowed for tf.zeros:

TypeError                                 Traceback (most recent call last)
<ipython-input-24-277eca38a392> in <module>()
      2 
      3 x = tf.placeholder("float", shape=[None, 4])
----> 4 y = tf.zeros([None, 4], "float")
      5 xy_stacked = tf.concat(1, [x, y])
      6 
[...]

/usr/local/lib/python3.4/dist-packages/numpy/core/_methods.py in _prod(a, axis, dtype, out, keepdims)
     33 
     34 def _prod(a, axis=None, dtype=None, out=None, keepdims=False):
---> 35     return umr_prod(a, axis, dtype, out, keepdims)
     36 
     37 def _any(a, axis=None, dtype=None, out=None, keepdims=False):

TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

I have figured out that it does not produce an error if I set the first dimension of my zeros tensor to non-None, such as 1:

x = tf.placeholder("float", shape=[None, 4])
y = tf.zeros([1, 4], "float")
xy_stacked = tf.concat(1, [x, y])

but then the resulting xy_stacked tensor is truncated to this size:

TensorShape([Dimension(None), Dimension(4)])
TensorShape([Dimension(1), Dimension(4)])
TensorShape([Dimension(1), Dimension(8)])

How can I pad the placeholder tensor with zeros so I get a tensor of shape TensorShape([Dimension(None), Dimension(8)]) in this example?

The only "solutions" I found so far is either something like the following:

x = tf.placeholder("float", shape=[None, 4])
y = 0 * x
xy_stacked = tf.concat(1, [x, y])

Or simply declaring y as a placeholder and always passing a zero array of the right size.

But neither looks like a clean solution to the problem and hacks like that get out of hand quickly in an application more complex than this simple example..

I'm using tensorflow-0.6.0-py3.

Answer 1

The recommended way to make a zero tensor with the same shape as another tensor is to use the tf.zeros_like() op:

x = tf.placeholder(tf.float32, shape=[None, 4])
y = tf.zeros_like(x)

The resulting tensor y appears to have the shape [None, None] according to Tensor.get_shape(), but at runtime it will expand to the same shape as x:

print y.get_shape()
# ==> TensorShape([Dimension(None), Dimension(None)])

sess = tf.Session()
y_result = sess.run(y, feed_dict={x: np.random.rand(4, 4)})

print y_result.shape
# ==> (4, 4)

The [None, None] static shape is returned because shape inference hasn't been specialized for tf.zeros_like(). I've filed a GitHub issue for that and it should be fixed soon.

---

EDIT: In your comment, you asked how to deal with the case where the zero tensor had a shape based on, but different from the original tensor. This is also possible, using tf.shape() and tf.stack() to build the dimensions, and tf.fill() to produce the zero tensor:

x = tf.placeholder(tf.float32, shape=[None, 4])

# Use tf.shape() to get the runtime size of `x` in the 0th dimension.
zeros_dims = tf.stack([tf.shape(x)[0], 7])

y = tf.fill(zeros_dims, 0.0)

sess = tf.Session()
y_result = sess.run(y, feed_dict={x: np.random.rand(4, 4)})
print y_result.shape
# ==> (4, 7)