How to load pre-loaded tensorflow constants into a model

Question

I do all of my tensorflow data generation AND training on the GPU. This means that my data sets are represented as tensorflow constants. For example:

import tensorflow as tf

# Generate x and y on the GPU
x = tf.constant([[1], [2], [3], [4]], dtype=tf.float32)
y_true = tf.constant([[0], [-1], [-2], [-3]], dtype=tf.float32)

This means that when I go to train my models, it looks something like the following code. I have greatly simplified it so that I can recreate my problem on other computers (based on the code at the bottom of the tensorflow low-level API tutorial Where it says "Complete Program")

import tensorflow as tf

# Declare constants
x = tf.constant([[1], [2], [3], [4]], dtype=tf.float32)
y_true = tf.constant([[0], [-1], [-2], [-3]], dtype=tf.float32)

# Create model to train
linear_model = tf.layers.Dense(units=1)
y_pred = linear_model(x) # <---- my model
loss = tf.losses.mean_squared_error(labels=y_true, predictions=y_pred)
optimizer = tf.train.AdamOptimizer(.2)
train = optimizer.minimize(loss)

# Boilerplate code
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)

# Train the model to give the correct y given the x
for i in range(100):
  _, loss_value = sess.run((train, loss))
  print(loss_value, end=", ")

# Test our output to make sure it looks good
print()
print(sess.run(y_pred))

# Generate new data to test or train in the model
new_x = tf.constant([[0], [1], [2]], dtype=tf.float32)
new_y_true = tf.constant([[10], [11], [12]], dtype=tf.float32)

# Test the model on new_x and see if it looks similar to new_y_true and/or train based on new_x and new_y_true
# ??

When I finish training my model (where y_pred is declared) I want to reuse that linear model and test it on my new_x and new_y_true data or even train on it. How would I do this?

I tried replacing x and y_true with placeholders, but you cannot put tf.Tensor objects into the feed_dict parameter.

score 0 · Answer 1 · answered Oct 24 '18 at 19:34

I'm not exactly sure why you want to define all your data in graph, and if you want to do this with more data than given in this toy example, you're going to run into problems. You can't feed a placeholder with a tensor (see Tensorflow: How to feed a placeholder variable with a tensor?).

The only thing that I can think of that would make this work is to use tf.cond with a boolean placeholder for indicating whether you want to train or test. Keep in mind that tf.cond is slow, and you should read it's documentation because it has some unintuitive behavior regarding which operations will be executed. It then looks like this:

import tensorflow as tf

# Declare constants
x = tf.constant([[1], [2], [3], [4]], dtype=tf.float32)
y_true = tf.constant([[0], [-1], [-2], [-3]], dtype=tf.float32)

new_x = tf.constant([[0], [1], [2]], dtype=tf.float32)
new_y_true = tf.constant([[10], [11], [12]], dtype=tf.float32)

# Define indicator if you want to train or test
is_train = tf.placeholder(dtype=tf.bool, shape=[], name="is_training")



# Define which input data to use depending if you are training or not
input_data = tf.cond(pred=is_train,
                 true_fn=lambda: x,
                 false_fn=lambda: new_x
)

# Create model to train
linear_model = tf.layers.Dense(units=1)
y_pred = linear_model(input_data)

loss = tf.losses.mean_squared_error(labels=y_true, predictions=y_pred)
optimizer = tf.train.AdamOptimizer(.2)
train =  optimizer.minimize(loss)

# Boilerplate code
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)

# Train the model to give the correct y given the x
for i in range(100):
  _, loss_value = sess.run((train, loss), feed_dict={is_train: True})
  print(loss_value, end=", ")

# Test our output to make sure it looks good
print()
print(sess.run(y_pred, feed_dict={is_train: False}))

I am building a reinforcement learning model, and in order to keep my code fast I want to avoid moving code to CPU and back to GPU. Speed is critical in my code, and that pipeline (CPU->GPU) is incredibly slow. So I need a way to declare my constants on the GPU, manipulate them on the GPU, and then run them through the model (or train the model on them). Is there a better way to use constants that exist on the GPU than using tensors? — Paul Terwilliger, Oct 25 '18 at 13:30
My code goes through a cycle where it manipulates GPU constants and then trains on them, then manipulates again, then trains... hundreds or thousands of times. I appreciate the effort with `tf.cond`, but it doesn't solve my problem because this solution only deals with 2 variables, where in my example I will have hundreds or more. — Paul Terwilliger, Oct 25 '18 at 13:33
Ok, I don't think this is something that you would normally do, because the graph structure is fixed. You will not be able to test the model on new data. Another approach that I imagine might work is to create a `linear_model` operation (with shared weights) for each possible input, but then you need to keep track of when to run what yourself. This will also be super memory-inefficient. — RunOrVeith, Oct 25 '18 at 13:43
You should pay the cost of data transfer if you want to do anything with the graph later, see also [How not to program the tensorflow graph](https://www.kdnuggets.com/2017/05/how-not-program-tensorflow-graph.html), in particular the paragraph "Avoid constants in the graph". With enough data you will also run into the [GraphDef 2GB limit](https://stackoverflow.com/questions/36349049/overcome-graphdef-cannot-be-larger-than-2gb-in-tensorflow). I suggest you build a fast data pipeline, e.g. with [GPU prefetching](https://www.tensorflow.org/api_docs/python/tf/contrib/data/prefetch_to_device) — RunOrVeith, Oct 25 '18 at 13:52

How to load pre-loaded tensorflow constants into a model

1 Answers1