41

I used sklearn for calculating TFIDF (Term frequency inverse document frequency) values for documents using command as :

from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(documents)
from sklearn.feature_extraction.text import TfidfTransformer
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)

X_train_tf is a scipy.sparse matrix of shape (2257, 35788).

How can I get TF-IDF for words in a particular document? More specific, how to get words with maximum TF-IDF values in a given document?

GIRISH kuniyal
  • 740
  • 1
  • 5
  • 14
maximus
  • 608
  • 3
  • 7
  • 17

3 Answers3

81

You can use TfidfVectorizer from sklean

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
from scipy.sparse.csr import csr_matrix #need this if you want to save tfidf_matrix

tf = TfidfVectorizer(input='filename', analyzer='word', ngram_range=(1,6),
                     min_df = 0, stop_words = 'english', sublinear_tf=True)
tfidf_matrix =  tf.fit_transform(corpus)

The above tfidf_matix has the TF-IDF values of all the documents in the corpus. This is a big sparse matrix. Now,

feature_names = tf.get_feature_names()

this gives you the list of all the tokens or n-grams or words. For the first document in your corpus,

doc = 0
feature_index = tfidf_matrix[doc,:].nonzero()[1]
tfidf_scores = zip(feature_index, [tfidf_matrix[doc, x] for x in feature_index])

Lets print them,

for w, s in [(feature_names[i], s) for (i, s) in tfidf_scores]:
  print w, s
sud_
  • 978
  • 8
  • 12
  • How do I get the words with the maximum tf-idf score? This is working for me, but I don't entirely understand what's going on in the last line. – tigerninjaman Nov 25 '18 at 16:30
  • 1
    [tfidf_matrix[doc, x] for x in feature_index] gives you the list of scores. I have zipped them with it's index so that you can pull the feature_name (or the word). You can sort the tfidf_scores based on the 2nd element (which is the score here) to get the maximum element first. – sud_ Nov 25 '18 at 21:36
  • May I know how to print for corresponding filenames ? – SVK Aug 23 '19 at 15:29
  • tfidf_matrix is a sparse matrix and each row is a document, e.g., in the answer I printed the tf-idf values of the 0th document. If you keep an array of document names, you can access it by the same index. – sud_ Aug 24 '19 at 18:25
  • why [doc,:] and not just [doc] ? is it not just slicing the whole row at index doc? – kg_sYy Apr 03 '20 at 18:41
  • It is, as the question seeks to find the tf-idf values of the words in a document. – sud_ Apr 20 '20 at 03:19
31

Here is another simpler solution in Python 3 with pandas library

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

vect = TfidfVectorizer()
tfidf_matrix = vect.fit_transform(documents)
df = pd.DataFrame(tfidf_matrix.toarray(), columns = vect.get_feature_names())
print(df)
8cold8hot
  • 431
  • 4
  • 7
  • 1
    I like it. Far more intuitive than having to remember the myriad "1 liner magical shortcuts" of Python – Sau001 May 16 '20 at 12:10
5

Finding tfidf score per word in a sentence can help in doing downstream task like search and semantics matching.

We can we get dictionary where word as key and tfidf_score as value.

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(min_df=3)
tfidf.fit(list(subject_sentences.values()))
feature_names = tfidf.get_feature_names()

Now we can write the transformation logic like this

def get_ifidf_for_words(text):
    tfidf_matrix= tfidf.transform([text]).todense()
    feature_index = tfidf_matrix[0,:].nonzero()[1]
    tfidf_scores = zip([feature_names[i] for i in feature_index], [tfidf_matrix[0, x] for x in feature_index])
    return dict(tfidf_scores)

E.g. For a input 

text = "increase post character limit"
get_ifidf_for_words(text)

output would be

{
'character': 0.5478868741621505,
'increase': 0.5487092618866405,
'limit': 0.5329156819959756,
'post': 0.33873144956352985
}
anshu kumar
  • 737
  • 7
  • 5