Tensorflow, compare an indexed value in a tensor with an integer for if condition

Question

I am using TensorFlow to do a customized embedding training similar to continuous bag of words (CBOW) model. However, unlike 'CBOW', which has a fixed length sliding window, my sliding window can be considered as flexible. Here is the problem:

Let's say, the embedding is word embedding. For word t, I have a tensor showing indexes of its context words: [-1, 1, 2, -1]. The maximum window size is 4, so the length of the vector is 4. But sometimes I do not have 4 context words for a word, so I use '-1' to mean 'no word in this position', and other integers are the index of a word. I also have an 'embedding' tensor, which is the embeddings for all the words.

What I am trying to do is to get the average embedding for the context words in order to represent the context. For example, if the context words are [-1, 1, 2, -1], I would get (1 * (embedding for word 1) + 2 * (embedding for word 2) ) / 2. I just need to neglect all the -1.

So in my code, I try to loop through the context word tensor to compare each value with -1 and use an if condition to control if I would add the embedding of this context word. I tried different ways for this, but always get 'TypeError: Using a tf.Tensor as a Python bool is not allowed.'

Is there a way to solve this problem? Or even better, is there a better representation of positions with no words so I can compute more efficiently (Tried to use NaN but also get a lot of troubles...)?

Thanks a lot for the help and hopefully I have a clear description of the problem.

Answer 1

I think it will be helpful to understand how to get values from the Tensorflow tensor. Take a look on the accepted answer here. Remember that Tensorflow may be seen as a symbolic language, so tf.Tensor is a symbol which tales certain values while evaluating within a session, not when it is defined or stack with other operations.

If you want to make a comparison within a graph, use tensorflow special functions for tensors comparisons like tf.equal, tf.less etc. For instace, I compare the first element of a tensor a with another constant tensor b:

# Here is an example of two tensors
a = tf.constant([[1.0, 0.0], [0.0, 1.0], [1.0, 0.0], [0.0, 1.0]])
b = tf.constant(1, dtype=tf.float32)

# generate a tensor representing a result of comparison
c = tf.equal(a[0, 0], b)
# Evaluate the output tensor in a session
print(tf.Session().run(c))

The output is True

Also, you may try to extract values of the tensor as follows:

# Here is an example of the tensor
a = tf.constant([[1.0, 0.0], [0.0, 1.0], [1.0, 0.0], [0.0, 1.0]])

# Evaluate the tensor in a session
print(tf.Session().run(a))

What is done here is conversion of the tf tensor into an numpy array, which you may process in any desired way.