Stack Overflow
Stack Overflow

Questions tagged [perplexity]

Perplexity is a measurement of how well a probability distribution or probability model predicts a sample.

From Wikipedia

In information theory, perplexity is a measurement of how well a probability distribution or probability model predicts a sample. It may be used to compare probability models. A low perplexity indicates the probability distribution is good at predicting the sample.

40 questions
6
votes
0 answers

Gensim lda gives negative log-perplexity value - is it normal and how can i interpret it?

I am currently using Gensim LDA for topic modeling. While Tuning hyper-parameters I found out that the model always gives negative log-perplexity Is it normal for model to behave like this?? (is it even possible?) if it is, is smaller perplexity…
nowheretogo
  • 125
  • 1
  • 5
6
votes
2 answers

Where is perplexity calculated in the Huggingface gpt2 language model code?

I see some github comments saying the output of the model() call's loss is in the form of perplexity: https://github.com/huggingface/transformers/issues/473 But when I look at the relevant…
user947659
  • 2,485
  • 4
  • 21
  • 24
6
votes
1 answer

How to interpret Sklearn LDA perplexity score. Why it always increase as number of topics increase?

I try to find the optimal number of topics using LDA model of sklearn. To do this I calculate perplexity by referring code on https://gist.github.com/tmylk/b71bf7d3ec2f203bfce2. But when I increase the number of topics, perplexity always increase …
JonghoKim
  • 1,965
  • 7
  • 21
  • 44
5
votes
0 answers

Is there a particular range for good perplexity value in NLP?

I'm fine-tuning a language model and am calculating training and validation losses along with the training and validation perplexities. It s calculated by taking the exponential of the loss, in my program. I'm aware that lower perplexities represent…
Dilrukshi Perera
  • 917
  • 3
  • 17
  • 31
5
votes
0 answers

Perplexity increases with number of topics

There are quite some posts about this specific issue, but I was unable to solve this problem. I have been experimenting with LDA on the 20newgroup corpus with both the Sklearn and Gensim implementation. It is described in the literature that…
Bas
  • 111
  • 8
5
votes
4 answers

How to compute perplexity using KenLM?

Let's say we build a model on this: $ wget https://gist.githubusercontent.com/alvations/1c1b388456dc3760ffb487ce950712ac/raw/86cdf7de279a2b9bceeb3adb481e42691d12fbba/something.txt $ lmplz -o 5 < something.txt > something.arpa From the perplexity…
alvas
  • 115,346
  • 109
  • 446
  • 738
3
votes
0 answers

Diagnostics (perplexity, LogLik, etc) for LDA topic model with textmodel_seededLDA package in R

I'm using the seededLDA package to do an LDA topic model. However, all of the packages and functions I've found to compute perplexity, log likelihood, exclusivity, etc (and other diagnostic tools) don't work on these models (they only work on…
Daniel Casey
  • 53
  • 3
3
votes
0 answers

t-sne perplexity for small data set

I am using t-SNE to visualize cytometry data. Most of guides I found (https://distill.pub/2016/misread-tsne/) warn how the choice of perplexity hyperparameter can influence the result. However, my data set size is really small, always expecting…
ludgo
  • 465
  • 1
  • 6
  • 13
2
votes
0 answers

Why am I randomly getting super high perplexities?

A few days ago I finished writing a word prediction program that tests both LSTM and GRU models on a given dataset. I test 4 models - 2 LSTM models and 2 GRU models. I wrote the program on Google Colab. I use two validation sets to see how it…
Guy
  • 155
  • 11
2
votes
0 answers

Assessing LDA predictions with textmineR in R - Calculating perplexity?

I am working on a LDA model with textmineR, have calculated coherence, log-likelihood measures and optimized my model. As a last step I would like to see how well the model predicts topics on unseen data. Thus, I am using the predict() function from…
lole_emily
  • 95
  • 9
2
votes
1 answer

How do i measure perplexity scores on a LDA model made with the textmineR package in R?

I've made a LDA topic model in R, using the textmineR package, it looks as follows. ## get textmineR dtm dtm2 <- CreateDtm(doc_vec = dat2$fulltext, # character vector of documents ngram_window = c(1, 2), doc_names…
Gustav Skov
  • 45
  • 7
2
votes
1 answer

Getting an error while executing perplexity function to evaluate the LDA model

I am trying to evaluate the topic modeling(LDA). Getting a error while execting perplexity function as: Error in (function (classes, fdef, mtable) : unable to find an inherited method for function ‘perplexity’ for signature ‘"LDA_Gibbs",…
Sachin
  • 21
  • 1
2
votes
1 answer

Determining log_perplexity using ldamulticore for optimum number of topics

I am trying to determine the optimum number of topics for my LDA model using log perplexity in python. That is, I am graphing the log perplexity for a range of topics and determining the minimum perplexity. However, the graph I have obtained has…
user8222177
  • 21
  • 2
1
vote
1 answer

Why is perplexity calculation giving different results for the same input?

I'm following Huggingface doc on calculating the perplexity of fixed-length models. I'm trying to verify that the formula works for various strings and I'm getting odd behavior. In particular, they mention We don’t want the log-likelihood for the…
Penguin
  • 1,923
  • 3
  • 21
  • 51
1
vote
0 answers

Large Language Model Perplexity

i am currently using GPT-3 and i am trying to compare its capabilities to related language models for my masters thesis. Unfortunatly GPT-3 is an API based application, so i am not really able to extract metrics such as perplexity. Over the API i…
Fabian
  • 63
  • 1
  • 13
1
2 3