8

I've just trained an LSTM language model using pytorch. The main body of the class is this: 

class LM(nn.Module):
    def __init__(self, n_vocab, 
                       seq_size, 
                       embedding_size, 
                       lstm_size, 
                       pretrained_embed):

        super(LM, self).__init__()
        self.seq_size = seq_size
        self.lstm_size = lstm_size
        self.embedding = nn.Embedding.from_pretrained(pretrained_embed, freeze = True)
        self.lstm = nn.LSTM(embedding_size,
                            lstm_size,
                            batch_first=True)
        self.fc = nn.Linear(lstm_size, n_vocab)

    def forward(self, x, prev_state):
        embed = self.embedding(x)
        output, state = self.lstm(embed, prev_state)
        logits = self.fc(output)

        return logits, state

Now I want to write a function which calculates how good a sentence is, based on the trained language model (some score like perplexity, etc.).

I'm a bit confused and I don't know how should I calculate this.
A similar sample would be of greate use.

MBT
  • 21,733
  • 19
  • 84
  • 102
P.Alipoor
  • 178
  • 1
  • 2
  • 11

1 Answers1

16

When using Cross-Entropy loss you just use the exponential function torch.exp() calculate perplexity from your loss.
(pytorch cross-entropy also uses the exponential function resp. log_n)

So here is just some dummy example:

import torch
import torch.nn.functional as F
num_classes = 10
batch_size  = 1

# your model outputs / logits
output      = torch.rand(batch_size, num_classes) 

# your targets
target      = torch.randint(num_classes, (batch_size,))

# getting loss using cross entropy
loss        = F.cross_entropy(output, target)

# calculating perplexity
perplexity  = torch.exp(loss)
print('Loss:', loss, 'PP:', perplexity)

In my case the output is:

Loss: tensor(2.7935) PP: tensor(16.3376)

You just need to be beware of that if you want to get the per-word-perplexity you need to have per word loss as well.

Here is a neat example for a language model that might be interesting to look at that also computes the perplexity from the output:

https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/language_model/main.py#L30-L50

MBT
  • 21,733
  • 19
  • 84
  • 102
  • Thank you for your answer. My question a bit differs, as I want to give just a sentence(a list of tokens) as an input, and get a score as output. Should I, in this case, give the sentence and the shifted-sentence as the _output_ and the _target_ in your sample code? – P.Alipoor Dec 07 '19 at 07:37
  • 1
    @P.Alipoor Yes, when looking at token with index **i** the target should be token **i+1**. – MBT Dec 07 '19 at 07:40