Compose a python array of all but every 17th element

Question

I have a question which is backwards of what most people seem to want. I'm working with numpy arrays, and I have to deal with a hardware bug in our data acquisition system. My data comes in as pairs of uint16s, which need to be split into 2 arrays. Right now, this is what I'm doing:

ql = open('/dev/quicklogic', 'rb')
self.rawData = numpy.fromfile(ql, numpy.uint16, 50000) # grab 50k integers
self.rawFlour = self.rawData[0::2] # deinterlace the array
self.rawScatter = self.rawData[1::2]

So, that works well to split up the array into the two data streams, however the device is sending over a pair of 0's every 17th read.

What's the most efficient way to construct a pair of arrays using everything except every 17th read? (Note, I could also just apply this to the rawData array instead of the two split arrays)

This might help http://stackoverflow.com/questions/509211/pythons-slice-notation — maxywb, May 05 '14 at 18:54
is `/dev/quicklogic` actually a file? or some kind of stream? I'm reluctant to believe that any array slice solutions will be the *efficient* answer you're after... — will, May 05 '14 at 19:16
@keyser Yah, the fact that it's coming from `/dev/quicklogic` makes me think it's actually some kind of input mounted in `/dev/` instead. like `/dev/random`. like [this](http://en.wikipedia.org/wiki/Special_file). Which would make me think it's being read from over and over again, and the 50K is just a buffering thing. — will, May 05 '14 at 19:27
@will, you nailed it. `/dev/quicklogic` is buffer filled by the kernel. The device samples every 200us, and the kernel fills a buffer which is returned to user space every 5 seconds. The processing needs to finish in under 5 seconds so I'm ready for the next round. The buffering, etc is all handled in a kernel driver I wrote about 6 months ago. — Maxwell Bottiger, May 05 '14 at 19:37
@MaxwellBottiger - Is it your kernel driver that's causing the 17th write to the file? My presumption would be that your bug is caused by [this](http://stackoverflow.com/a/3774467/432913) quirk of integers and pointers, in your driver code. — will, May 05 '14 at 19:44
Did you mean that the buffer of 50k needs to be cleared in 5s? A This part will take closer to 1μs :p — keyser, May 06 '14 at 09:43
@will - That's a really interesting little piece about integers. I'm going to read the followup article later today. Although a good guess, we've confirmed it is an actual hardware bug. We've put the FPGA on a scope and sure enough, it's there every time. — Maxwell Bottiger, May 08 '14 at 12:26
@MaxwellBottiger - huh. That's really odd. And that was my thought on the article when i first read it. I always find the articles on the more esoteric optimisations made quite interesting. What goes on under the hood, so to speak. — will, May 08 '14 at 17:55

score 2 · Accepted Answer · answered May 05 '14 at 19:01

Let's suppose your array size is a multiple of 17. You can use reshape:

import numpy as np

a = np.arange(17*2)
print a
"""
[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
 25 26 27 28 29 30 31 32 33]
"""
a = a.reshape(a.size // 17, 17)
print a
"""
[[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16]
 [17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33]]
"""
a = a[:, :-1]
print a
"""
[[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15]
 [17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32]]
"""
a = a.ravel()
print a
"""
[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 17 18 19 20 21 22 23 24 25
 26 27 28 29 30 31 32]
"""

Or maybe `arange`? `a = np.delete(a, np.arange(0, a.size, 17))` — devnull, May 05 '14 at 19:04

score 2 · Answer 2 · answered May 05 '14 at 19:21

2

You can do this with list slicing by using the whole array and retroactivly deleting every 17th element.

>>> x = range(10)
>>> del x[::4]  
>>> x
[1, 2, 3, 5, 6, 7, 9]

As you can see this deletes every 4th element starting with element 0. If you want to keep the first element you can do this:

>>> x = range(10)
>>> del x[3::4] 
>>> x
[0, 1, 2, 4, 5, 6, 8, 9]

answered May 05 '14 at 19:21

maxywb

2,275
1
19
25

I forgot about that one. +1 – keyser May 05 '14 at 19:22
1

The equivalent numpy construct is `np.delete(np.arange(10), slice(None, None, 4))` or `np.delete(np.arange(10), slice(3, None, 4))`. – Jaime May 05 '14 at 20:24

A.J. Uppal · Answer 3 · 2014-05-05T19:11:55.943

You can use list comprehension as follows:

>>> n = 17
>>> mylist = range(1, 100)
>>> new_list = [mylist[i] for i in range(len(mylist)) if i % n != n-1]
>>> new_list
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
>>>

This removes every n^th item in the range of (1, 100), basically all numbers starting with 1 and ending with 99.

keyser · Answer 4 · 2014-05-05T19:13:55.650

Here's one using list comprehension (i is the index):

>>> a = range(1,100)
>>> a = [x for x in (x for i,x in enumerate(a) if (i+1)%17 != 0)]
>>> a
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]

The basic idea is to skip every 17th i, which is accomplished by skipping indexes that fulfill (i+1)%17, since python uses 0-indexing.

Compose a python array of all but every 17th element

4 Answers4