I recently need to initialize a large embedding matrix in tensorflow (greater than 2GB) and the size hits the 2GB limit on protocol buffer.
I searched stackoverflow for help and was able to fine this excellent answer explaining the use of placeholder for this kind of situation. While the placeholder solution works quite well, I found out that the following solution using tf initializer also works, but I don't understand why this solution will not be a problem to the 2GB limit.
import tensorflow as tf
mb = 2 ** 20
gb = 2 ** 30
tf.reset_default_graph()
with tf.Session() as sess:
value = tf.random_uniform([2 * gb]) # using np.random.randn will break the limit
v = tf.get_variable("v", initializer=value, dtype=tf.float32)
sess.run(tf.global_variables_initializer())
print(sess.run(v[:10]))
graph = tf.get_default_graph()
print(graph.as_graph_def().ByteSize() / mb) # 0.0012083053588867188
Does this work because tf.random* initializers internally inspect the shape of the tensors to be generated and switch to a placeholder implementation accordingly? However, if we look at the graph on tensorboard, we will not see any placeholder node in the graph.
Thanks!
