How to check if a key exists in a word2vec trained model or not

Question

I have trained a word2vec model using a corpus of documents with Gensim. Once the model is training, I am writing the following piece of code to get the raw feature vector of a word say "view".

myModel["view"]

However, I get a KeyError for the word which is probably because this doesn't exist as a key in the list of keys indexed by word2vec. How can I check if a key exits in the index before trying to get the raw feature vector?

score 46 · Answer 1 · edited Jul 19 '18 at 19:22

46

Word2Vec also provides a 'vocab' member, which you can access directly.

Using a pythonistic approach:

if word in w2v_model.vocab:
    # Do something

EDIT Since gensim release 2.0, the API for Word2Vec changed. To access the vocabulary you should now use this:

if word in w2v_model.wv.vocab:
    # Do something

EDIT 2 The attribute 'wv' is being deprecated and will be completed removed in gensim 4.0.0. Now it's back to the original answer by OP:

if word in w2v_model.vocab:
    # Do something

edited Jul 19 '18 at 19:22

tomar__

105
5

answered Jun 29 '15 at 10:02

Matt Fortier

1,213
1
10
18

1

@MohitJain And how does that not provider an answer to his question? I think my answer makes perfect sense, considering this is exactly the code I use myself to solve this problem. – Matt Fortier Jun 30 '15 at 01:52
1

One liners without explanations are more suitable for comments. You can garner more upvotes if you can add brief explanation for readers like me. – Mohit Jain Jun 30 '15 at 02:21
Thanks for explanation, noted! – Matt Fortier Jun 30 '15 at 03:14
If using *gensim* use rather `gensim.models.Word2Vec.wv.vocab` as indicated by rakaT below – Titou May 16 '17 at 15:18
@Titou Yes, the interface for gensim Word2Vec changed. Thanks for pointing it out! – Matt Fortier May 17 '17 at 02:59
How fast is this? Would it make sense to build a set from the w2v.vocab for faster access? – Geoffrey Negiar Jul 24 '18 at 21:16

score 35 · Accepted Answer · edited Mar 19 '17 at 12:45

35

convert the model into vectors with

word_vectors = model.wv

then we can use

if 'word' in word_vectors.vocab

edited Mar 19 '17 at 12:45

pacholik

8,607
9
43
55

answered Mar 19 '17 at 12:27

rakaT

474
4
2

quemeful · Answer 3 · 2022-11-28T19:10:37.263

14

The vocab attribute was removed from KeyedVector in Gensim 4.0.0. Try using this:

if 'word' in model.wv.key_to_index:
    # do something

https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4#4-vocab-dict-became-key_to_index-for-looking-up-a-keys-integer-index-or-get_vecattr-and-set_vecattr-for-other-per-key-attributes

edited Nov 28 '22 at 19:10

answered Nov 08 '21 at 17:58

quemeful

9,542
4
60
69

score 1 · Answer 4 · answered May 18 '15 at 11:33

1

Answering my own question here.

Word2Vec provides a method named contains('view') which returns True or False based on whether the corresponding word has been indexed or not.

answered May 18 '15 at 11:33

London guy

27,522
44
121
179

8

For future reference, this doesn't work anomore. `'Word2Vec' object has no attribute 'contains'` – CentAu Dec 13 '15 at 23:32

score 1 · Answer 5 · answered Mar 09 '18 at 05:23

1

I generally use a filter:

for doc in labeled_corpus:
    words = filter(lambda x: x in model.vocab, doc.words)

This is one simple method for getting past the KeyError on unseen words.

answered Mar 09 '18 at 05:23

Prakhar Agarwal

2,724
28
31

score 1 · Answer 6 · answered Aug 14 '22 at 12:32

1

as @quemeful has mentioned, you could do something like,

if "view" in model.wv.key_to_index.keys():
    # do something

answered Aug 14 '22 at 12:32

Ishara Dissanayake

74
6

score 0 · Answer 7 · answered Mar 21 '16 at 16:45

Hey i know am getting late this post, but here is a piece of code that can handle this issue well. I myself using it in my code and it works like a charm :)

   size = 300 #word vector size
   word = 'food' #word token

   try:
        wordVector = model[word].reshape((1, size))
   except KeyError:
        print "not found! ",  word

NOTE: I am using python Gensim Library for word2vec models

score 0 · Answer 8 · answered Oct 19 '22 at 19:56

to check if the word is exist in your model you can use

word2vec_pretrained_dict = dict(zip(w2v_model.key_to_index.keys(), w2v_model.vectors))

where w2v_model.key_to_index give you dictionary of each word and sequance number

and w2v_model.vectors return the vectorized for of each word

How to check if a key exists in a word2vec trained model or not

8 Answers8