44

I have read LDA and I understand the mathematics of how the topics are generated when one inputs a collection of documents.

References say that LDA is an algorithm which, given a collection of documents and nothing more (no supervision needed), can uncover the “topics” expressed by documents in that collection. Thus by using LDA algorithm and the Gibbs Sampler (or Variational Bayes), I can input a set of documents and as output I can get the topics. Each topic is a set of terms with assigned probabilities.

What I don't understand is, if the above is true, then why do many topic modeling tutorials talk about separating the dataset into training and test set?

Can anyone explain me the steps (the basic concept) of how LDA can be used for training a model, which can then be used to analyze another test dataset?

lambda
  • 3,295
  • 1
  • 26
  • 32
tan
  • 1,569
  • 5
  • 14
  • 30

2 Answers2

40

Splitting the data into training and testing sets is a common step in evaluating the performance of a learning algorithm. It's more clear-cut for supervised learning, wherein you train the model on the training set, then see how well its classifications on the test set match the true class labels. For unsupervised learning, such evaluation is a little trickier. In the case of topic modeling, a common measure of performance is perplexity. You train the model (like LDA) on the training set, and then you see how "perplexed" the model is on the testing set. More specifically, you measure how well the word counts of the test documents are represented by the word distributions represented by the topics.

Perplexity is good for relative comparisons between models or parameter settings, but it's numeric value doesn't really mean much. I prefer to evaluate topic models using the following, somewhat manual, evaluation process:

  1. Inspect the topics: Look at the highest-likelihood words in each topic. Do they sound like they form a cohesive "topic" or just some random group of words?
  2. Inspect the topic assignments: Hold out a few random documents from training and see what topics LDA assigns to them. Manually inspect the documents and the top words in the assigned topics. Does it look like the topics really describe what the documents are actually talking about?

I realize that this process isn't as nice and quantitative as one might like, but to be honest, the applications of topic models are rarely quantitative either. I suggest evaluating your topic model according to the problem you're applying it to.

Good luck!

gregamis
  • 411
  • 3
  • 3
  • Thanks gregamis for the nice explanation. For your point number 2). topic assignments to document, how do you go about assigning topics to words in a document? Do you sequentially go through the words in the document and lookup for the same word in a topic and assign that topic to that word? Well if you do that, what happens when a word is present in more than 1 topic with high probability? Say a very trivial example, 'river bank' and 'bank account'. – tan Jun 26 '12 at 18:57
  • 1
    @tan: The topics should be assigned to documents not to words. Also not manually. – Sai Manoj Kumar Yadlapati Jan 09 '14 at 11:56
  • @gregamis : Thank you for this post. So, do we use supervised LDA/ any other topic modelling technique to evaluate the topic models only? – pnv Apr 27 '15 at 07:27
  • Possibly you could use a similarity measure etc cosine on an ideal set vs the results of the test set. – Jonathan Apr 20 '16 at 09:39
  • For a working example concerning the use of the perplexity measure in cross validation to define an optimum number of topics for an LDA model you might refer to this answer: [link](https://stackoverflow.com/questions/21355156/topic-models-cross-validation-with-loglikelihood-or-perplexity) – Manuel Bickel Oct 31 '17 at 14:53
  • Has the train/test split to be made on document basis or on mere randomised split (e.g. usal 70/30 rule, on the whole cleaned doc corpus?) – CheTesta May 28 '18 at 12:15
0

The general rule that using the training data for evaluation might be subject to overfitting also applies to unsupervised learning like LDA -- though it is not as obvious. LDA optimizes some objective, ie. generative probability, on the training data. It might be that in the training data two words are indicative of a topic, say "white house" for US politics. Assume the two words only occur once (in the training data). Then any algorithm fully relying on the assumption that they indicate only politics and nothing else would be doing great if you evaluated on the training data. However, if there are other topics like "architecture" then you might question, whether this is really the right thing to learn. Having a test data set can solve this issue to some extend:

  1. Since the relationship "white house" seems rare in the training data, it likely does not occur at all in the test data. If so, the evaluation shows how much your model relies on spurious relationships that might in fact not be helpful compared to more general ones.
  2. "White house" occurs in the test data, say it occurs once for "US politics" and once in a document on architecture. Then the assumption that it only indicates "US politics" is too strong and performance metrics will be worse, showing that your model is overfitting.
J.T.
  • 194
  • 5