counting n-gram frequency in python nltk

Question

I have the following code. I know that I can use apply_freq_filter function to filter out collocations that are less than a frequency count. However, I don't know how to get the frequencies of all the n-gram tuples (in my case bi-gram) in a document, before I decide what frequency to set for filtering. As you can see I am using the nltk collocations class.

import nltk
from nltk.collocations import *
line = ""
open_file = open('a_text_file','r')
for val in open_file:
    line += val
tokens = line.split()

bigram_measures = nltk.collocations.BigramAssocMeasures()
finder = BigramCollocationFinder.from_words(tokens)
finder.apply_freq_filter(3)
print finder.nbest(bigram_measures.pmi, 100)

Have you tried `finder.ngram_fd.viewitems()`? – Suzana Jan 16 '13 at 20:07 — Suzana, Jan 16 '13 at 20:07
Thanks finder.ngram_fd.viewitems() works! – Rkz Jan 21 '13 at 01:21 — Rkz, Jan 21 '13 at 01:21

Ram Narasimhan · Answer 1 · 2013-11-15T18:25:03.487

48

NLTK comes with its own bigrams generator, as well as a convenient FreqDist() function.

f = open('a_text_file')
raw = f.read()

tokens = nltk.word_tokenize(raw)

#Create your bigrams
bgs = nltk.bigrams(tokens)

#compute frequency distribution for all the bigrams in the text
fdist = nltk.FreqDist(bgs)
for k,v in fdist.items():
    print k,v

Once you have access to the BiGrams and the frequency distributions, you can filter according to your needs.

Hope that helps.

edited Nov 15 '13 at 18:25

answered Jan 19 '13 at 10:05

Ram Narasimhan

22,341
5
49
55

This leaves me with `File "/usr/local/lib/python3.6/site-packages/nltk/util.py", line 467, in ngrams while n > 1: TypeError: '>' not supported between instances of 'str' and 'int'` – m02ph3u5 Nov 23 '17 at 15:21

score 12 · Accepted Answer · answered Jan 21 '13 at 01:22

12

The finder.ngram_fd.viewitems() function works

answered Jan 21 '13 at 01:22

Rkz

1,237
5
16
30

1

seems like that was deprecated, but you can use `finder.ngram_fd[('this', 'bigram')]` to get frequency of a bigram – wordsforthewise Dec 12 '18 at 17:37

score 7 · Answer 3 · answered Nov 01 '19 at 09:11

I tried all the above and found a simpler solution. NLTK comes with a simple Most Common freq Ngrams.

filtered_sentence is my word tokens

import nltk
from nltk.util import ngrams
from nltk.collocations import BigramCollocationFinder
from nltk.metrics import BigramAssocMeasures

word_fd = nltk.FreqDist(filtered_sentence)
bigram_fd = nltk.FreqDist(nltk.bigrams(filtered_sentence))

bigram_fd.most_common()

This should give the output as:

[(('working', 'hours'), 31),
 (('9', 'hours'), 14),
 (('place', 'work'), 13),
 (('reduce', 'working'), 11),
 (('improve', 'experience'), 9)]

score 6 · Answer 4 · answered Mar 08 '18 at 18:02

6

from nltk import FreqDist
from nltk.util import ngrams    
def compute_freq():
   textfile = open('corpus.txt','r')

   bigramfdist = FreqDist()
   threeramfdist = FreqDist()

   for line in textfile:
        if len(line) > 1:
        tokens = line.strip().split(' ')

        bigrams = ngrams(tokens, 2)
        bigramfdist.update(bigrams)
compute_freq()

answered Mar 08 '18 at 18:02

Vahab

307
4
7

just insert indents after 'if'; code works if python 3.5 – Vahab Mar 08 '18 at 18:04

counting n-gram frequency in python nltk

4 Answers4

Linked