I am trying to understand the difference between a placeholder and a variable in TensorFlow:
X = tf.placeholder("float")
W = tf.Variable(rng.randn(), name="weight")
I also read the Stack Overflow's question below. I understand the difference when they are the input of a model.
InvalidArgumentError: You must feed a value for placeholder tensor Placeholder
However, in general, if we are not building a model, is there still a difference between tf.placeholder() and tf.Variable()?
A placeholder is used for feeding external data into a Tensorflow computation (stuff outside the graph). Here's some documentation: (https://www.tensorflow.org/versions/r0.10/how_tos/reading_data/#feeding)
TensorFlow's feed mechanism lets you inject data into any Tensor in a computation graph. A python computation can thus feed data directly into the graph.
I personally would draw an analogy from placeholders to reading from standard input.
x = raw_input()
X = tf.placeholder("float")
When you read from standard input, you need to "inject data" from an external source. Same with a placeholder. It lets you "inject data" that's external to the computation graph.
If you're training a learning algorithm, the clear use case of placeholder is to feed in your training data. The training data isn't stored in the computation graph. How are you going to get it into the graph? By injecting it through a placeholder. A placeholder is basically you telling the graph "I don't have this for you yet. But I'll have it for you when I ask you to run."
A variable is used to store state in your graph. It requires an initial value. One use case could be representing weights of a neural network or something similar. Here's documentation: (https://www.tensorflow.org/api_docs/python/tf/Variable)
A variable maintains state in the graph across calls to run(). You add a variable to the graph by constructing an instance of the class Variable.
The Variable() constructor requires an initial value for the variable, which can be a Tensor of any type and shape. The initial value defines the type and shape of the variable. After construction, the type and shape of the variable are fixed. The value can be changed using one of the assign methods.
I personally would draw an analogy between Tensorflow Variables and assigning a variable in Python to anything that is not dependent on external stuff. For example,
# Tensorflow:
W = tf.Variable(rng.randn(), name="weight")
# Standard python:
w = 5
w = "hello"
w = [1, 2, 3, 4, 5]
W represents some sort of result of your computation. Just like how you must initialize all your variables in Python (you can't just run a command x you have to say x = ...something...), you have to initialize all Variable objects in Tensorflow.
There's not much related between tf.Variable and tf.placeholder in my opinion. You use a Variable if you need to store state. You use a placeholder if you need to input external data.
If you are not building a model, you should still use tf.placeholder if you want to insert external data that you don't necessarily have while you're defining the graph. If you are not building a model, you still need tf.Variable if you want to store some kind of result of your computation while the graph is being run.
I'm not an expert in Tensorflow, so I can only speculate as to why the design has both.
A big difference between placeholders and variables is that placeholders can have variable size, but the shape of a tf.Variable must be specified while constructing the graph.
Variable size placeholders sense: maybe I only want to input a training batch of size 5 right now, but maybe I want to increase the batch size later on. Maybe I don't know ahead of time how many training examples I'm going to get.
Variable size variables don't make sense: tf.Variable holds the learned parameters of your model, and the number of parameters shouldn't change. Furthermore, Tensorflow extends to distributed computation. If you had Variables whose shape changed throughout the computation, it would be very difficult to keep it properly distributed among 1000 computers.
Usually, you build a model and all parameters are known ahead of time, so that's what tf.Variable is probably used to represent. tf.placeholder is probably for everything else outside of your model (or computation graph) and so that can be more flexible.
The most obvious difference between the tf.Variable and the tf.placeholder is that
you use variables to hold and update parameters. Variables are in-memory buffers containing tensors. They must be explicitly initialized and can be saved to disk during and after training. You can later restore saved values to exercise or analyze the model.
Initialization of the variables is done with sess.run(tf.global_variables_initializer()). Also while creating a variable, you need to pass a Tensor as its initial value to the Variable() constructor and when you create a variable you always know its shape.
On the other hand you can't update the placeholder. They also should not be initialized, but because they are a promise to have a tensor, you need to feed the value into them sess.run(<op>, {a: <some_val>}). And at last, in comparison to a variable, placeholder might not know the shape. You can either provide parts of the dimensions or provide nothing at all.
There other differences:
Interesting part is that not only placeholders can be fed. You can feed the value to a Variable and even to a constant.
