Cannot feed pre-train word embedding before training

Question

I want to load pre-train word embedding before training instead of loading it every train_steps. I follow the step in this post. But it will show error:

You must feed a value for placeholder tensor 'word_embedding_placeholder' with dtype float and shape [2000002,300]

Here is the roughly code:

embeddings_var = tf.Variable(tf.random_uniform([vocabulary_size, embedding_dim], -1.0, 1.0), trainable=False)
embedding_placeholder = tf.placeholder(tf.float32, [vocabulary_size, embedding_dim], name='word_embedding_placeholder')
embedding_init = embeddings_var.assign(embedding_placeholder)  # assign exist word embeddings
batch_embedded = tf.nn.embedding_lookup(embedding_init, batch_ph)


sess = tf.Session()
train_steps = round(len(X_train) / BATCH_SIZE)

train_iterator, train_next_element = get_dataset_iterator(X_train, y_train, BATCH_SIZE, training_epochs)

sess.run(init_g)
sess.run(train_iterator.initializer)
_ = sess.run(embedding_init, feed_dict={embedding_placeholder: w2v})

for epoch in range(0, training_epochs):
    # Training steps
    for i in range(train_steps):
        X_train_input, y_train_input = sess.run(train_next_element)            
        seq_len = np.array([list(word_idx).index(PADDING_INDEX) if PADDING_INDEX in word_idx else len(word_idx) for word_idx in X_train_input])  # actual lengths of sequences
        train_loss, train_acc, _ = sess.run([loss, accuracy, optimizer],
                                   feed_dict={batch_ph: X_train_input,
                                              target_ph: y_train_input,
                                              seq_len_ph: seq_len,
                                              keep_prob_ph: KEEP_PROB})

When I change feed_dict in training to:

train_loss, train_acc, _ = sess.run([loss, accuracy, optimizer],
                                   feed_dict={batch_ph: X_train_input,
                                              target_ph: y_train_input,
                                              seq_len_ph: seq_len,
                                              keep_prob_ph: KEEP_PROB,
                                              embedding_placeholder: w2v})

it works, but it is not elegant. Does anyone meet this issue?

Goal: I want to load the pre-train embedding only once before training. Instead of recomputing embedding_init every time.

score 0 · Answer 1 · answered Feb 18 '19 at 08:55

0

Presumably you use batch_embedded somewhere in your network, which means it is used in your loss. That means that whenever you perform sess.run on loss inside the loop, you are recomputing batch_embedded, and hence recomputing embedding_init, for which you need the embedding_placeholder. Instead you can initialise your variable as follows:

embeddings_var = tf.get_variable("embeddings_var", shape=[vocabulary_size, embedding_dim], initializer=tf.constant_initializer(w2v),  trainable=False)

answered Feb 18 '19 at 08:55

tomkot

926
5
7

yeah, I used this method before, but the embedding size is bigger than 2GB. So this method is not working. – Jim Feb 18 '19 at 09:43
Are you trying to fine tune the weights? You current version does not do that, since it recomputes embedding_init every time. If you indeed do not want to fine tune the weights, another approach could be to have batch_embedded be a placeholder and do the embedding lookup in numpy. – tomkot Feb 18 '19 at 10:16
No, I just want to train the model so I set embedding trainable=False. Do you have some related post for do the embedding lookup in numpy? – Jim Feb 18 '19 at 10:20
I don't, but I think you can just use indexing like this: if you have for example `batch_indices = [204,189]`, you can get the corresponding embeddings with `batch_embeddings = w2v[batch_indices, :]` – tomkot Feb 18 '19 at 10:28

Cannot feed pre-train word embedding before training

1 Answers1