Deleting multiple lists with two conditions in python

Question

I have two lists containing some image pixel values and its "measurements". I'm trying to delete items from those if it satisfies two conditions. I tried looping over it with a for loop before realizing my dumb mistake. My code is below. What could be a method to do it?

for i in range(100):
    delete_chance = np.random.random()
    if abs(measurements[i]) == 0.15 and delete_chance < 0.70:
        del images[i]
        del measurements[i]

What's the dumb mistake? Why doesn't this code work and what should it do instead? — Nils Werner, May 12 '17 at 13:34
If you just iterate over the `reversed(range(100))` instead it can actually be safe. — wim, May 12 '17 at 13:36
Whatever solution you might end up using, you should eliminate the float equal comparison. There are special function for float comparison, look it up. — Liran Funaro, May 12 '17 at 13:46

score 2 · Accepted Answer · edited May 23 '17 at 11:54

2

A list comprehension can help:

filtered_images, filtered_measurements = zip(*[(i, m) for i, m in zip(images, measurements) if math.isclose(abs(m), 0.15) and random.random() < 0.7])

Note we are using math.isclose() which has only been added to Python 3.5. If you're on older versions you'll have to write your own isclose().

If speed is an issue (thousands or millions of images) you can use NumPy, too:

images = numpy.array(images)
measurements = numpy.array(measurements)

mask = numpy.logical_and(
    numpy.random.rand(images.shape[0]) < 0.7,
    numpy.isclose(numpy.abs(measurements), 0.15)
)

filtered_measurements = measurements[mask]
filtered_images = images[mask]

edited May 23 '17 at 11:54

Community

1
1

answered May 12 '17 at 13:39

Nils Werner

34,832
7
76
98

Almost got it where I want. But this seems to, while downsampling randomly, mask all the other values. I would like to keep those values. How would I go about doing that? – prat_pad May 12 '17 at 14:27
I don't understand what your're asking – Nils Werner May 12 '17 at 14:30
Your numpy code produces an output that masks all the values other than abs(0.15). I want to reduce the number of abs(0.15) values, but I would like to keep all the other values which were present in the list. – prat_pad May 12 '17 at 14:32
You mean `removed_images = images[~mask]`? – Nils Werner May 12 '17 at 14:36
Bingo! Thank you – prat_pad May 12 '17 at 14:42

M. Shaw · Answer 2 · 2017-05-12T13:44:14.630

0

Try filter:

z = zip(*list(filter(lambda x: x[1] != 0.15 or np.random.random() >= 0.7, zip(measurements, images))))
measurements = list(z[0])
images = list(z[1])

edited May 12 '17 at 13:44

answered May 12 '17 at 13:35

M. Shaw

1,742
11
15

You're using values from `images` as an index for `measurements`. I don't think this will work. Instead try zipping `images` and `measurements`. – Nils Werner May 12 '17 at 13:36

Liran Funaro · Answer 3 · 2017-05-12T14:24:48.317

First, you should never compare two floats directly, you should compare them using a function that accept a reasonable error (epsilon).

Second, first build an iterator that return True for whatever element you want to remove:

t = [np.random.random() < 0.7 and np.isclose(abs(measurements[i]), 0.15) for i in range(100)]

Then build a new list accordingly:

measurments = [m for m,b in zip(measurments,t) if not b]
images = [m for m,b in zip(images,t) if not b)]

score -1 · Answer 4 · answered May 12 '17 at 13:37

-1

You could iterate in reversed order:

for i in range(99, -1, -1):
    delete_chance = np.random.random()
    if abs(measurements[i]) == 0.15 and delete_chance < 0.70:
        del images[i]
        del measurements[i]

answered May 12 '17 at 13:37

Mike Scotty

10,530
5
38
50

Deleting multiple lists with two conditions in python

4 Answers4