4

I am working on implementing a Python script for NMF text data clustering. In my work I am using Scikit NMF implementation, however as I understand, in Scikit NMF is more like classification method than a clustering method.

I have developed a simple script working on some sample arcticles. I am preprocessing them and putting as an input for NMF. According to papers shared by my professor I receive some clusters, however I have no idea how to visualize/present them.

Anyone of you have any idea how to make this human-friendly to read? :)

Code of main script below:

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.cluster import KMeans
from sklearn.preprocessing import normalize
from sklearn.decomposition import LatentDirichletAllocation, NMF
from sklearn.preprocessing import normalize
import pandas as pd
from preprocess import *


# loading data
raw_text_data = loading_bbc_datasets(5)

text_data = text_preparing(raw_text_data)
tf_vectorizer = TfidfVectorizer()
Y = tf_vectorizer.fit_transform(text_data)
Y_norm = normalize(Y)

nmf = NMF(n_components=5, random_state=1, alpha=.1, l1_ratio=0.5)
A = nmf.fit_transform(Y_norm)
X = nmf.components_
features = tf_vectorizer.get_feature_names()
print(features)

AF = pd.DataFrame(Y_norm.toarray())
WF = pd.DataFrame(A)
HF = pd.DataFrame(X)

AF.to_csv('Y.csv', sep=',', header=features)
WF.to_csv('A.csv', sep=',', header=['C1', 'C2', 'C3', 'C4', 'C5'])
HF.to_csv('X.csv', sep=',', header=features)
rafmat24
  • 83
  • 2
  • 8

1 Answers1

4

NMF is not a classification method, it is a dimensionality reduction method. When you process your texts with CountVectorizer, you have a high number of dimensions and NMF allows to reduce it.

NMF approximates the documents*terms matrix X by: W * H.

The new dimensions can be seen as topics, so for a given document you can see in W which topics the document belongs the most since the values are non negative (if the value is high, the document is highly related to the topic).

Similarly, for a given topic, you can get the words highly related to it using H.

To answer your question, you can cluster the documents into topics and represent each topic in a human-friendly manner by giving the most related words.

Stanislas Morbieu
  • 1,721
  • 7
  • 11
  • 2
    Dimensionality reduction is the classic use case. But, worth mentioning that NMF is also a clustering method. Under certain constraints, it is theoretically equivalent to K-Means. – MattY Jan 04 '20 at 09:18