0

My code should output the top 10 words with the highest frequency in the corpus. However, it is giving the output of 10 random words. 

from nltk.corpus import brown
import operator

brown_tagged_sentences = brown.tagged_sents(categories='news')
fd=nltk.FreqDist(brown.words(categories='news'))
sorted_fd = dict(sorted(fd.items(), key=operator.itemgetter(1), reverse=True))
print(sorted_fd)
most_freq_words=list(sorted_fd)[:10]
for word in most_freq_words:
    print(word,':',sorted_fd[word])

The output coming currently is below which is wrong:

Rae : 1
discharge : 1
ignition : 1
contendere : 1
done : 24
meaning : 4
ashore : 1
Francesca : 1
Vietnamese : 1
data : 4

Kindly help

Bonson
  • 1,418
  • 4
  • 18
  • 38
  • 1
    Dictionaries are unsorted data structures, they don't preserve order, read more in the [Documentation](https://docs.python.org/3/tutorial/datastructures.html). – DjaouadNM Sep 10 '17 at 11:45
  • I was looking at this and trying to implement this. But it isnt working. Not sure if I have got it right. https://stackoverflow.com/a/613218/1181744 – Bonson Sep 10 '17 at 11:47

2 Answers2

2

The nltk's FreqDist() class can directly give you its contents in descending order of frequency, using the method most_common():

fd=nltk.FreqDist(brown.words(categories='news'))
for w, f in fd.most_common(10):
    print(w+":", f)
alexis
  • 48,685
  • 16
  • 101
  • 161
0

Got a work around. Hope this is the best way od doing it:

fd=nltk.FreqDist(brown.words(categories='news'))
sorted_fd = sorted(fd.items(), key=operator.itemgetter(1), reverse=True)
most_freq_words = [w for w,c in sorted_fd]
most_freq_words = most_freq_words[:10]
for word in most_freq_words:
    print(word,':',fd[word])
Bonson
  • 1,418
  • 4
  • 18
  • 38
  • 1
    The best "workaround" is [`OrderedDict`](https://docs.python.org/3/library/collections.html#collections.OrderedDict). – Right leg Sep 10 '17 at 12:03