How to train Word Embeddings in Tensorflow

Question

So I have changed the model described here to perform multi-class text classification instead of a binary class classification. http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/

My model is overfitting even after applying L2 regularization so I am wanting to use a pre-trained word2vec model. But I am extremely new to Tensorflow & deep learning & am not sure where to start.

Code: https://github.com/dennybritz/cnn-text-classification-tf/blob/master/text_cnn.py#L27

Here is the relevant code that I want to change to use the Google pre-trained word2vec model:

        # Embedding layer
    with tf.device('/cpu:0'), tf.name_scope("embedding"):
        W = tf.Variable(
            tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),
            name="W")
        self.embedded_chars = tf.nn.embedding_lookup(W, self.input_x)
        self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)

It will be very helpful if someone can point me to how can i incorporate this in the code. I looked at the embedding_lookup doc but that doesn't seem to have the information i am looking for.

score 1 · Answer 1 · edited May 23 '17 at 12:32

1

Refere to these links:

edited May 23 '17 at 12:32

Community

1
1

answered Oct 08 '16 at 20:43

Jules G.M.

3,624
1
21
35

I have already seen the 1st post and made the following change with no luck: `model = word2vec.Word2Vec.load_word2vec_format('./word2vec/pre-trained/GoogleNews-vectors-negative300.bin', binary=True) embedding = model.syn0 vocab_size = embedding.shape[0] embedding_dim = embedding.shape[1] ` . But the above gives me the following message: `UserWarning: Converting sparse IndexedSlices to a dense Tensor with 900000000 elements. This may consume a large amount of memory` and the training seems to be hung at this step – Rookie Oct 08 '16 at 21:15

score 0 · Answer 2 · answered Jul 13 '18 at 05:55

  W = tf.Variable(
        tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),
        name="W")

This line created a word dic randomly to represent the word in your own vocab, So you just need load the pre-trained word2vec dic to replace your 'W', it will works

How to train Word Embeddings in Tensorflow

2 Answers2