17

What are symbolic tensors in TensorFlow and Keras? How are they different than other tensors? Why do they even exist? Where do they come up in TensorFlow and Keras? How should we deal with them or what problems can we face when dealing with them?

In the past, I had faced certain issues related to symbolic tensors, such as the _SymbolicException, but the documentation does not describe this concept. There's also another post where this question is also asked, but, in this post, I am focusing on this specific question, so that answers can be later used as a reference.

nbro
  • 15,395
  • 32
  • 113
  • 196

1 Answers1

13

According to blog.tensorflow.org, a symbolic tensor differs from other tensors in that they do not specifically hold values.

Let's consider a simple example.

>>> a = tf.Variable(5, name="a")
>>> b = tf.Variable(7, name="b")
>>> c = (b**2 - a**3)**5
>>> print(c)

The output is as follows:

tf.Tensor(1759441920, shape=(), dtype=int32)

For the above, the values are specifically defined in tf.Variable format, and the output is in Tensor format. However, the tensor must contain a value in order to be considered as such.

Symbolic tensors are different in that no explicit values are required to define the tensor, and this has implications in terms of building neural networks with TensorFlow 2.0, which now uses Keras as the default API.

Here is an example of a Sequential neural network that is used to build a classification model for predicting hotel cancellation incidences (full Jupyter Notebook here if interested):

from tensorflow.keras import models
from tensorflow.keras import layers

model = models.Sequential()
model.add(layers.Dense(8, activation='relu', input_shape=(4,)))
model.add(layers.Dense(1, activation='sigmoid'))

This is a symbolically defined model, as no values are explicitly being defined in the network. Rather, a framework is created for the input variables to be read by the network, and then generate predictions.

In this regard, Keras has become quite popular given that it allows for building of graphs using symbolic tensors, while at the same time maintaining an imperative layout.

Michael Grogan
  • 973
  • 5
  • 10
  • 4
    You say "This is a symbolically defined model". How is this related to symbolic tensors? Yes, it may seem obvious, but I would like a precise answer. – nbro Jan 12 '20 at 19:55
  • 1
    Certainly. It is related in the sense that a symbolic tensor does not require a pre-defined value, and as such can take on a whole host of values. Therefore, a symbolically defined model is one where, by definition, the tensors are symbolic. If the tensors were not symbolic, then it would not be possible to run the neural network without explicitly defining the values for the input variables in the first instance. – Michael Grogan Jan 12 '20 at 20:09
  • I have two (or three) more questions. 1. Is there a way to programmatically understand whether a tensor is symbolic or not (akin to `type(my_var)` in Python for getting the type of a variable)? 2. I've read the blog post you linked to, where it is mentioned that symbolic tensors come up when building models with the sequential and functional APIs. Are there any other cases where symbolic tensors can come up? 3. What about TF's placeholders? How are placeholders different from symbolic tensors, or why aren't symbolic tensors called placeholders? – nbro Jan 12 '20 at 23:45
  • 1
    1. According to the tensorflow.org website, "A Tensor is a symbolic handle to one of the outputs of an Operation. It does not hold the values of that operation's output, but instead provides a means of computing those values in a TensorFlow tf.compat.v1.Session." It's not simply a case of symbolic vs. not symbolic - the purpose of a Tensor in the first place is to provide a framework to evaluate operations. 2. Yes, but the situation described would be one of the most frequent. 3. Placeholders are not compatible with eager execution, hence why the two aren't interchangeable. – Michael Grogan Jan 13 '20 at 20:13
  • How would you extract the value of one of these symbolic tensor once the model is training? For example the output of a NN will be a symbolic tensor and therefore have no attribute ```.numpy()``` – ScubaNinjaDog Aug 20 '22 at 14:10