3

How do I measure or find the Zipf distribution ? For example, I have a corpus of english words. How do I find the Zipf distribution ? I need to find the Zipf ditribution and then plot a graph of it. But I am stuck in the first step which is to find the Zipf distribution.

Edit: From the frequency count of each word, it is clear that it obeys the Zipf law. But my aim is to plot a zipf distribution graph. I have no idea about how to calculate the data for the distribution graph

Anil_M
  • 10,893
  • 6
  • 47
  • 74
RDM
  • 1,136
  • 3
  • 28
  • 50
  • 1
    is this your homework assignment? Please show us what have you tried? – kmario23 Apr 28 '17 at 03:29
  • No this isn't my homework assignment. This is more of a hobby project. I am analysing an ancient script called Indus Script . More details here: (http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0009506). The script consists of symbols and not words. I first translated the corpus of symbols into a sequence of numbers and have performed number of analysis on it. For the zipf distribution, I have calculated the frequency of each symbol and I dont know how to proceed from there on. – RDM Apr 28 '17 at 03:33
  • If your goal is ONLY to plot it, just plot a histogram of the counts: http://matplotlib.org/1.2.1/examples/api/histogram_demo.html – Josep Valls Apr 28 '17 at 03:39
  • 1
    No, I need to plot a zipf distribution graph to show that the data in the corpus obeys Zipf law. From the frequency count, I can clearly see that it does obey the Zipf law, but then I should be able to fit it on the Zipf distribution graph. – RDM Apr 28 '17 at 03:40

1 Answers1

7

I don't pretend to understand statistics. However, based upon reading from scipy site, here is a naive attempt in python.

Build Data

First we get our data. For example we download data from National Library of Medicine MeSH (Medical Subject Heading) ASCII file d2016.bin (28 MB).
Next, we open file, convert to string.

open_file = open('d2016.bin', 'r')
file_to_string = open_file.read()

Next we locate individual words in the file and separate out words.

words = re.findall(r'(\b[A-Za-z][a-z]{2,9}\b)', file_to_string)

Finally we prepare a dict with unique words as key and word count as values.

for word in words:
    count = frequency.get(word,0)
    frequency[word] = count + 1

Build zipf distribution data
For speed purpose we limit data to 1000 words.

n = 1000
frequency = {key:value for key,value in frequency.items()[0:n]}

After that we get frequency of values , convert to numpy array and use numpy.random.zipf function to draw samples from a zipf distribution.

Distribution parameter a =2. as a sample as it needs to be greater than 1. For visibility purpose we limit data to 50 sample points.

s = frequency.values()
s = np.array(s)

count, bins, ignored = plt.hist(s[s<50], 50, normed=True)
x = np.arange(1., 50.)
y = x**(-a) / special.zetac(a)

And finally plot the data.

Putting All Together 

import re
from operator import itemgetter
import matplotlib.pyplot as plt
from scipy import special
import numpy as np

#Get our corpus of medical words
frequency = {}
open_file = open('d2016.bin', 'r')
file_to_string = open_file.read()
words = re.findall(r'(\b[A-Za-z][a-z]{2,9}\b)', file_to_string)

#build dict of words based on frequency
for word in words:
    count = frequency.get(word,0)
    frequency[word] = count + 1

#limit words to 1000
n = 1000
frequency = {key:value for key,value in frequency.items()[0:n]}

#convert value of frequency to numpy array
s = frequency.values()
s = np.array(s)

#Calculate zipf and plot the data
a = 2. #  distribution parameter
count, bins, ignored = plt.hist(s[s<50], 50, normed=True)
x = np.arange(1., 50.)
y = x**(-a) / special.zetac(a)
plt.plot(x, y/max(y), linewidth=2, color='r')
plt.show()

Plot
enter image description here

Anil_M
  • 10,893
  • 6
  • 47
  • 74
  • Thank you for the detailed explanation. I am facing an error at:`count, bins, ignored = plt.hist(s[s<50], 50, normed=True) TypeError: unorderable types: dict_values() < int()` I am using Python 3.5.2, however I faced the same error on Python 2.x. Will try to fix it and see how it goes. – RDM Apr 28 '17 at 05:09
  • Awesome, this worked out. Thanks a ton for the help. Really appreciate your help. – RDM Apr 28 '17 at 05:17
  • This answer lacks many variables. The line y = x**(-a) / special.zetac(a) does not compile, as special is not defined. Furthermore you never use numpy.random.zipf despite mentioning it. If you could update this answer with all the variables and steps included it would be great. – TheForgot3n1 Sep 29 '20 at 17:04
  • Hi There, The solution still works for me using python 2.7. For python 3.x it may need some changes. special.zetac(a) is having static value as 0.6449340668482264 – Anil_M Dec 19 '20 at 08:02