Python histogram from unsorted data

Question

I am trying to parse some data to generate a histogram

The data is in multiple columns but the only relevant column for me are the two below.

X

AB    42

CD    77

AB    33

AB    42

AB    33

CD    54

AB    33

Only for the rows with AB, I want to plot the histogram of the value in col 2. So the histogram should sort and plot:

33 - 3

42 - 2

(even though 42 occurs first, I want to plot 33 first).

I have a lot of columns but it needs to grep the 'AB' character and only search in those rows. Can anyone help?

UPDATE: Data is in a csv file and there are several columns.

EDIT: I now have the data in a csv file in this format.

Addresses,Data

FromAP,42

FromAP,42

FromAP,33

ToAP,77

FromAP,54

FromAP,42

FromAP,42

FromAP,33

ToAP,42

FromAP,42

FromAP,33

If I use the code from @dranxo,

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('data.csv', sep=',')

df_useful = df[df['Addresses'] == 'FromAP']

df_useful.hist()
plt.show()

I get the following error:

Laptop@ubuntu:~/temp$ ./a.py
/usr/lib/pymodules/python2.7/matplotlib/axes.py:8261: UserWarning: 2D hist input should be nsamples x nvariables;
 this looks transposed (shape is 0 x 1)
  'this looks transposed (shape is %d x %d)' % x.shape[::-1])
Traceback (most recent call last):
  File "./a.py", line 11, in <module>
    df_useful.hist()
   File "/usr/lib/python2.7/dist-packages/pandas/tools/plotting.py", line 2075, in hist_frame
    ax.hist(data[col].dropna().values, **kwds)
  File "/usr/lib/pymodules/python2.7/matplotlib/axes.py", line 8312, in hist
    xmin = min(xmin, xi.min())
  File "/usr/lib/python2.7/dist-packages/numpy/core/_methods.py", line 21, in _amin
    out=out, keepdims=keepdims)
ValueError: zero-size array to reduction operation minimum which has no identity

I do have the pandas package, numpy, matplotlib installed. Thanks

Answer 1

The following code sample will work. Please note reading the CSV may be a little different in your sitatuion depending on its exact format. See this question for reading a CSV.

import csv
with open("/tmp/test.csv", "r") as f:
    #Filter the result for "AB" as we read the lines from the file
    filtered_result = [tuple(line) for line in csv.reader(f) if line[0] == "AB"]

#Now, sort the result by the second column
final_result = sorted(filtered_result,key=lambda x: x[1])

#Print it for inspection
for key, value in final_result:
    print "key: %s, value: %s" % (key, value)

Output:

key: AB, value: 33
key: AB, value: 33
key: AB, value: 33
key: AB, value: 42
key: AB, value: 42

Contents of /tmp/test.csv:

AB,42
CD,77
AB,33
AB,42
AB,33
CD,54
AB,33

I populated /tmp/test.csv with 100,000 lines of random data, and here is how long my script takes:

$ time python test.py 

real    0m0.073s
user    0m0.073s
sys 0m0.000s

Edit: Updated for better performance and to show example of CSV
Edit: Updated again to be even faster

Answer 2

There are two different problems:

1. Parse CSV - Python has an inbuilt library for CSV.
2. Graph your results - Does your Python program need to generate the histogram? Or is it acceptable to put your parsed CSV into some spreadsheet software and do it there?

If you have to have your Python program generate the histogram, then here's a list of graphing libraries to get you started.

Answer 3

I suppose data is in file.csv and AB is in first column and 42 is in second column

import csv
reader = csv.reader(open('file.csv', 'r'))
dic = {}
for row in reader:
    if row[0] == 'AB':
        value = int(row[1])
        if  value in dic.keys():
            dic[value] += 1
        else:
            dic[value] = 1

#sorted print 
for key in sorted(dic):
    print '%s-%s'%(key, dic[key])

Answer 4

Have you ever looked into pandas?

Here's how to parse the data and plot in a few lines:

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('data.ssv', sep=' ')

df_useful = df[df['letters'] == 'AB']

df_useful.hist()
plt.show()

Note: I saved your data into a file called 'data.ssv' before calling pd.read_csv. Here's that file:

letters numbers

AB 42

CD 77

AB 33

AB 42

AB 33

CD 54

AB 33

edit: To check that the problem isn't with the data you can run this code:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

df = pd.DataFrame(np.round(np.random.randn(10, 2)),
                 columns=['a', 'b'])

df.hist()
plt.show()