Histogram manipulation to remove unwanted data

Question

How do I remove data from a histogram in python under a certain frequency count?

Say I have 10 bins, the first bin has a count of 4, the second has 2, the third has 1, fourth has 5, etc... Now I want to get rid of the data that has a count of 2 or less. So the second bin would go to zero, as would the third.

Example:

import numpy as np
import matplotlib.pyplot as plt

gaussian_numbers = np.random.randn(1000)
plt.hist(gaussian_numbers, bins=12)
plt.title("Gaussian Histogram")
plt.xlabel("Value")
plt.ylabel("Frequency")

fig = plt.gcf()

Gives:

and I want to get rid of the bins with fewer than a frequency of say 'X' (could be frequency = 100 for example).

want:

thank you.

You don't use your import in the example, however you use np and plt, which I guess are from numpy and matplotlib.pyplot. — Jérôme, Mar 31 '16 at 21:21

score 4 · Accepted Answer · edited May 23 '17 at 10:28

4

Une np.histogram to create the histogram.

Then use np.where. Given a condition, it yields an array of booleans you can use to index your histogram.

import numpy as np
import matplotlib.pyplot as plt

gaussian_numbers = np.random.randn(1000)

# Get histogram
hist, bins = np.histogram(gaussian_numbers, bins=12)

# Threshold frequency
freq = 100

# Zero out low values
hist[np.where(hist <= freq)] = 0

# Plot
width = 0.7 * (bins[1] - bins[0])
center = (bins[:-1] + bins[1:]) / 2
plt.bar(center, hist, align='center', width=width)
plt.title("Gaussian Histogram")
plt.xlabel("Value")
plt.ylabel("Frequency")

(Plot part inspired from here.)

edited May 23 '17 at 10:28

Community

1
1

answered Mar 31 '16 at 19:19

Jérôme

13,328
7
56
106

no I want to get rid of the values IF the frequency count on that bin is low – sci-guy Mar 31 '16 at 19:25
You mean set the value to 0 or remove the value completely? What would you expect in my example for instance? – Jérôme Mar 31 '16 at 19:28
Yes get rid of them completely :) There is a reason to the madness – sci-guy Mar 31 '16 at 19:36
what is histog? Still having some issues implementing it into the above example. – sci-guy Mar 31 '16 at 19:40
ok Thank you - right now I am just getting an error that 'histog' is not defined – sci-guy Mar 31 '16 at 19:44
so what do I plot do get the amended histogram? – sci-guy Mar 31 '16 at 20:55
Yes, but now I loose my associated x data with each bin..... and ya- i was wrong, I dont want to remove the data point, just to make it zero – sci-guy Mar 31 '16 at 21:07
OK so you don't want small bins ripped out but zeroed. See edit. – Jérôme Mar 31 '16 at 21:18
ValueError: x and y must have same first dimension - histog_x is dimension 13 where histog_y is dimension 12 – sci-guy Mar 31 '16 at 21:27
I don't think you can plot it like that, Since histog_x is the bin edges and histog_y is the count, that is why histog_y is always dim: (histog_x - 1) -- it goes between the two numbers – sci-guy Mar 31 '16 at 21:35
Yes, sorry, I was editing. Hopefully this should work. – Jérôme Mar 31 '16 at 21:36

Histogram manipulation to remove unwanted data

1 Answers1

Linked