How to print the LDA topics models from gensim? Python

Question

Using gensim I was able to extract topics from a set of documents in LSA but how do I access the topics generated from the LDA models?

When printing the lda.print_topics(10) the code gave the following error because print_topics() return a NoneType:

Traceback (most recent call last):
  File "/home/alvas/workspace/XLINGTOP/xlingtop.py", line 93, in <module>
    for top in lda.print_topics(2):
TypeError: 'NoneType' object is not iterable

The code:

from gensim import corpora, models, similarities
from gensim.models import hdpmodel, ldamodel
from itertools import izip

documents = ["Human machine interface for lab abc computer applications",
              "A survey of user opinion of computer system response time",
              "The EPS user interface management system",
              "System and human system engineering testing of EPS",
              "Relation of user perceived response time to error measurement",
              "The generation of random binary unordered trees",
              "The intersection graph of paths in trees",
              "Graph minors IV Widths of trees and well quasi ordering",
              "Graph minors A survey"]

# remove common words and tokenize
stoplist = set('for a of the and to in'.split())
texts = [[word for word in document.lower().split() if word not in stoplist]
         for document in documents]

# remove words that appear only once
all_tokens = sum(texts, [])
tokens_once = set(word for word in set(all_tokens) if all_tokens.count(word) == 1)
texts = [[word for word in text if word not in tokens_once]
         for text in texts]

dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]

# I can print out the topics for LSA
lsi = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=2)
corpus_lsi = lsi[corpus]

for l,t in izip(corpus_lsi,corpus):
  print l,"#",t
print
for top in lsi.print_topics(2):
  print top

# I can print out the documents and which is the most probable topics for each doc.
lda = ldamodel.LdaModel(corpus, id2word=dictionary, num_topics=50)
corpus_lda = lda[corpus]

for l,t in izip(corpus_lda,corpus):
  print l,"#",t
print

# But I am unable to print out the topics, how should i do it?
for top in lda.print_topics(10):
  print top

Something is missing in your code, namely corpus_tfidf computation. Would you please add the remaining piece? — mel, Feb 05 '15 at 16:38
Your code can not be reproduced. Please update your code - you are missing of the code i.e. corpus_tfidf . — Sade, Sep 01 '23 at 08:38

score 21 · Answer 1 · answered Feb 22 '13 at 03:00

21

After some messing around, it seems like print_topics(numoftopics) for the ldamodel has some bug. So my workaround is to use print_topic(topicid):

>>> print lda.print_topics()
None
>>> for i in range(0, lda.num_topics-1):
>>>  print lda.print_topic(i)
0.083*response + 0.083*interface + 0.083*time + 0.083*human + 0.083*user + 0.083*survey + 0.083*computer + 0.083*eps + 0.083*trees + 0.083*system
...

answered Feb 22 '13 at 03:00

alvas

115,346
109
446
738

6

`print_topics` is an alias for `show_topics` with the first five topics. Just write `lda.show_topics()`, no `print` necessary. – mac389 Feb 24 '13 at 21:35
1

@alvas can you kindly explain what is the value in the output. For example 0.083 for the response; what does it mean in plain English. Thanks – adhg May 05 '19 at 00:56

score 12 · Answer 2 · answered Apr 29 '15 at 09:04

I think syntax of show_topics has changed over time:

show_topics(num_topics=10, num_words=10, log=False, formatted=True)

For num_topics number of topics, return num_words most significant words (10 words per topic, by default).

The topics are returned as a list – a list of strings if formatted is True, or a list of (probability, word) 2-tuples if False.

If log is True, also output this result to log.

Unlike LSA, there is no natural ordering between the topics in LDA. The returned num_topics <= self.num_topics subset of all topics is therefore arbitrary and may change between two LDA training runs.

score 9 · Answer 3 · answered Oct 22 '18 at 22:29

I think it is alway more helpful to see the topics as a list of words. The following code snippet helps acchieve that goal. I assume you already have an lda model called lda_model.

for index, topic in lda_model.show_topics(formatted=False, num_words= 30):
    print('Topic: {} \nWords: {}'.format(idx, [w[0] for w in topic]))

In the above code, I have decided to show the first 30 words belonging to each topic. For simplicity, I have shown the first topic I get.

Topic: 0 
Words: ['associate', 'incident', 'time', 'task', 'pain', 'amcare', 'work', 'ppe', 'train', 'proper', 'report', 'standard', 'pmv', 'level', 'perform', 'wear', 'date', 'factor', 'overtime', 'location', 'area', 'yes', 'new', 'treatment', 'start', 'stretch', 'assign', 'condition', 'participate', 'environmental']
Topic: 1 
Words: ['work', 'associate', 'cage', 'aid', 'shift', 'leave', 'area', 'eye', 'incident', 'aider', 'hit', 'pit', 'manager', 'return', 'start', 'continue', 'pick', 'call', 'come', 'right', 'take', 'report', 'lead', 'break', 'paramedic', 'receive', 'get', 'inform', 'room', 'head']

I don't really like the way the above topics look so I usually modify my code to as shown:

for idx, topic in lda_model.show_topics(formatted=False, num_words= 30):
    print('Topic: {} \nWords: {}'.format(idx, '|'.join([w[0] for w in topic])))

... and the output (first 2 topics shown) will look like.

Topic: 0 
Words: associate|incident|time|task|pain|amcare|work|ppe|train|proper|report|standard|pmv|level|perform|wear|date|factor|overtime|location|area|yes|new|treatment|start|stretch|assign|condition|participate|environmental
Topic: 1 
Words: work|associate|cage|aid|shift|leave|area|eye|incident|aider|hit|pit|manager|return|start|continue|pick|call|come|right|take|report|lead|break|paramedic|receive|get|inform|room|head

@alvas, I am sorry my answer is coming quite late but I would love to know what you think about it. — Samuel Nde, Oct 22 '18 at 22:31

score 6 · Answer 4 · answered Mar 04 '13 at 08:58

6

Are you using any logging? print_topics prints to the logfile as stated in the docs.

As @mac389 says, lda.show_topics() is the way to go to print to screen.

answered Mar 04 '13 at 08:58

zanbri

5,958
2
31
41

i'm not using any logging because i need to use the topics immediately. you're right, the `lda.show_topics()` or `lda.print_topic(i)` is the way to go. – alvas Mar 06 '13 at 23:40

score 5 · Answer 5 · answered Aug 28 '19 at 04:07

Using Gensim for cleaning it's own topic format.

from gensim.parsing.preprocessing import preprocess_string, strip_punctuation,
strip_numeric

lda_topics = lda.show_topics(num_words=5)

topics = []
filters = [lambda x: x.lower(), strip_punctuation, strip_numeric]

for topic in lda_topics:
    print(topic)
    topics.append(preprocess_string(topic[1], filters))

print(topics)

Output :

(0, '0.020*"business" + 0.018*"data" + 0.012*"experience" + 0.010*"learning" + 0.008*"analytics"')
(1, '0.027*"data" + 0.020*"experience" + 0.013*"business" + 0.010*"role" + 0.009*"science"')
(2, '0.026*"data" + 0.016*"experience" + 0.012*"learning" + 0.011*"machine" + 0.009*"business"')
(3, '0.028*"data" + 0.015*"analytics" + 0.015*"experience" + 0.008*"business" + 0.008*"skills"')
(4, '0.014*"data" + 0.009*"learning" + 0.009*"machine" + 0.009*"business" + 0.008*"experience"')


[
  ['business', 'data', 'experience', 'learning', 'analytics'], 
  ['data', 'experience', 'business', 'role', 'science'], 
  ['data', 'experience', 'learning', 'machine', 'business'], 
  ['data', 'analytics', 'experience', 'business', 'skills'], 
  ['data', 'learning', 'machine', 'business', 'experience']
]

Shirish Kumar · Answer 6 · 2015-02-10T18:22:48.970

Here is sample code to print topics:

def ExtractTopics(filename, numTopics=5):
    # filename is a pickle file where I have lists of lists containing bag of words
    texts = pickle.load(open(filename, "rb"))

    # generate dictionary
    dict = corpora.Dictionary(texts)

    # remove words with low freq.  3 is an arbitrary number I have picked here
    low_occerance_ids = [tokenid for tokenid, docfreq in dict.dfs.iteritems() if docfreq == 3]
    dict.filter_tokens(low_occerance_ids)
    dict.compactify()
    corpus = [dict.doc2bow(t) for t in texts]
    # Generate LDA Model
    lda = models.ldamodel.LdaModel(corpus, num_topics=numTopics)
    i = 0
    # We print the topics
    for topic in lda.show_topics(num_topics=numTopics, formatted=False, topn=20):
        i = i + 1
        print "Topic #" + str(i) + ":",
        for p, id in topic:
            print dict[int(id)],

        print ""

I tried to run your code where I pass list of list containing BOW to text. I get following error: TypeError: show_topics() got an unexpected keyword argument 'topics' — mribot, Feb 09 '15 at 21:31

score 3 · Answer 7 · answered Aug 01 '16 at 09:42

3

you can use:

for i in  lda_model.show_topics():
    print i[0], i[1]

answered Aug 01 '16 at 09:42

xu2mao

572
6
8

score 1 · Answer 8 · answered Oct 29 '17 at 21:33

Recently, came across a similar issue while working with Python 3 and Gensim 2.3.0. print_topics() and show_topics() weren't giving any error but also not printing anything. Turns out that show_topics() returns a list. So one can simply do:

topic_list = show_topics()
print(topic_list)

Feng Mai · Answer 9 · 2018-06-14T22:25:01.027

You can also export the top words from each topic to a csv file. topn controls how many words under each topic to export.

import pandas as pd

top_words_per_topic = []
for t in range(lda_model.num_topics):
    top_words_per_topic.extend([(t, ) + x for x in lda_model.show_topic(t, topn = 5)])

pd.DataFrame(top_words_per_topic, columns=['Topic', 'Word', 'P']).to_csv("top_words.csv")

The CSV file has the following format

Topic Word  P  
0     w1    0.004437  
0     w2    0.003553  
0     w3    0.002953  
0     w4    0.002866  
0     w5    0.008813  
1     w6    0.003393  
1     w7    0.003289  
1     w8    0.003197 
...

score 0 · Answer 10 · answered Feb 14 '19 at 06:18

****This code works fine but I want to know the topic name instead of Topic: 0 and Topic:1, How do i know which topic this word comes in**?** 



for index, topic in lda_model.show_topics(formatted=False, num_words= 30):
        print('Topic: {} \nWords: {}'.format(idx, [w[0] for w in topic]))

Topic: 0 
Words: ['associate', 'incident', 'time', 'task', 'pain', 'amcare', 'work', 'ppe', 'train', 'proper', 'report', 'standard', 'pmv', 'level', 'perform', 'wear', 'date', 'factor', 'overtime', 'location', 'area', 'yes', 'new', 'treatment', 'start', 'stretch', 'assign', 'condition', 'participate', 'environmental']
Topic: 1 
Words: ['work', 'associate', 'cage', 'aid', 'shift', 'leave', 'area', 'eye', 'incident', 'aider', 'hit', 'pit', 'manager', 'return', 'start', 'continue', 'pick', 'call', 'come', 'right', 'take', 'report', 'lead', 'break', 'paramedic', 'receive', 'get', 'inform', 'room', 'head']

LDA model doesn't define the topic on its own, depending on the words in that topic, you have to rename it. It just finds the most used words depending on the 'bag_of_words'. — aakarsh, Apr 08 '21 at 11:35

How to print the LDA topics models from gensim? Python

10 Answers10

Linked