Stack Overflow
Stack Overflow

Questions tagged [tf-idf]

“Term-frequency ⨉ Inverse Document Frequency”, or “tf-idf”, measures how important a word is to a document in a collection or corpus.

“Term-frequency ⨉ Inverse Document Frequency”, or “tf-idf”, in Natural Language Processing () and , measures how important a word is to a document in a collection or corpus.

References:

1326 questions
112
votes
6 answers

Python: tf-idf-cosine: to find document similarity

I was following a tutorial which was available at Part 1 & Part 2. Unfortunately the author didn't have the time for the final section which involved using cosine similarity to actually find the distance between two documents. I followed the…
add-semi-colons
  • 18,094
  • 55
  • 145
  • 232
67
votes
3 answers

TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document

I'm using TfidfVectorizer from scikit-learn to do some feature extraction from text data. I have a CSV file with a Score (can be +1 or -1) and a Review (text). I pulled this data into a DataFrame so I can run the Vectorizer. This is my code: import…
boltthrower
  • 1,230
  • 3
  • 12
  • 29
55
votes
5 answers

Simple implementation of N-Gram, tf-idf and Cosine similarity in Python

I need to compare documents stored in a DB and come up with a similarity score between 0 and 1. The method I need to use has to be very simple. Implementing a vanilla version of n-grams (where it possible to define how many grams to use), along…
seanieb
  • 1,196
  • 2
  • 14
  • 36
50
votes
3 answers

Scikit Learn TfidfVectorizer : How to get top n terms with highest tf-idf score

I am working on keyword extraction problem. Consider the very general case from sklearn.feature_extraction.text import TfidfVectorizer tfidf = TfidfVectorizer(tokenizer=tokenize, stop_words='english') t = """Two Travellers, walking in the noonday…
AbtPst
  • 7,778
  • 17
  • 91
  • 172
48
votes
5 answers

Why is log used when calculating term frequency weight and IDF, inverse document frequency?

The formula for IDF is log( N / df t ) instead of just N / df t. Where N = total documents in collection, and df t = document frequency of term t. Log is said to be used because it “dampens” the effect of IDF. What does this mean? Also, why do we…
stevetronix
  • 1,231
  • 2
  • 16
  • 32
45
votes
4 answers

How to get tfidf with pandas dataframe?

I want to calculate tf-idf from the documents below. I'm using python and pandas. import pandas as pd df = pd.DataFrame({'docId': [1,2,3], 'sent': ['This is the first sentence','This is the second sentence', 'This is the third…
user1610952
  • 1,249
  • 1
  • 16
  • 31
45
votes
4 answers

TFIDF for Large Dataset

I have a corpus which has around 8 million news articles, I need to get the TFIDF representation of them as a sparse matrix. I have been able to do that using scikit-learn for relatively lower number of samples, but I believe it can't be used for…
apurva.nandan
  • 1,061
  • 1
  • 11
  • 19
43
votes
3 answers

Can I use CountVectorizer in scikit-learn to count frequency of documents that were not used to extract the tokens?

I have been working with the CountVectorizer class in scikit-learn. I understand that if used in the manner shown below, the final output will consist of an array containing counts of features, or tokens. These tokens are extracted from a set of…
tumultous_rooster
  • 12,150
  • 32
  • 92
  • 149
42
votes
6 answers

Cosine similarity and tf-idf

I am confused by the following comment about TF-IDF and Cosine Similarity. I was reading up on both and then on wiki under Cosine Similarity I find this sentence "In case of of information retrieval, the cosine similarity of two documents will…
N00programmer
  • 1,111
  • 4
  • 13
  • 17
41
votes
1 answer

How to see top n entries of term-document matrix after tfidf in scikit-learn

I am new to scikit-learn, and I was using TfidfVectorizer to find the tfidf values of terms in a set of documents. I used the following code to obtain the same. vectorizer = TfidfVectorizer(stop_words=u'english',ngram_range=(1,5),lowercase=True) X =…
Amrith Krishna
  • 2,768
  • 3
  • 31
  • 65
37
votes
7 answers

How do I calculate the cosine similarity of two vectors?

How do I find the cosine similarity between vectors? I need to find the similarity to measure the relatedness between two lines of text. For example, I have two sentences like: system for user interface user interface machine … and their…
shiva
37
votes
1 answer

Using Sklearn's TfidfVectorizer transform

I am trying to get the tf-idf vector for a single document using Sklearn's TfidfVectorizer object. I create a vocabulary based on some training documents and use fit_transform to train the TfidfVectorizer. Then, I want to find the tf-idf vectors for…
Sterling
  • 3,835
  • 14
  • 48
  • 73
32
votes
2 answers

tf-idf feature weights using sklearn.feature_extraction.text.TfidfVectorizer

this page: http://scikit-learn.org/stable/modules/feature_extraction.html mentions: As tf–idf is a very often used for text features, there is also another class called TfidfVectorizer that combines all the option of CountVectorizer and…
fast tooth
  • 2,317
  • 4
  • 25
  • 34
27
votes
5 answers

Keep TFIDF result for predicting new content using Scikit for Python

I am using sklearn on Python to do some clustering. I've trained 200,000 data, and code below works well. corpus = open("token_from_xml.txt") vectorizer = CountVectorizer(decode_error="replace") transformer = TfidfTransformer() tfidf =…
lol.Wen
  • 812
  • 2
  • 9
  • 17
24
votes
5 answers

Interpreting the sum of TF-IDF scores of words across documents

First let's extract the TF-IDF scores per term per document: from gensim import corpora, models, similarities documents = ["Human machine interface for lab abc computer applications", "A survey of user opinion of computer system…
alvas
  • 115,346
  • 109
  • 446
  • 738
1
2 3
…
88 89