python, summing numpy arrays of same dimensions, different shapes with offset and rolling over the end

Question

I have 2 numpy arrays and want to sum them with offset. The sum always have shape of array "a"

a = np.array([1, 1, 1, 1, 1])
b = np.array([5, 6])
sumWithRoll(a, b, offset=1)
print(a)
>> [1, 6, 7, 1, 1]

Also if array "b" is long or offset is big enough it should roll over the end of array "a":

a = np.array([1, 1, 1, 1, 1])
b = np.array([5, 6, 7, 8])
sumWithRoll(a, b, offset=3)
print(a)
>> [8, 9, 1, 6, 7]

I need this for merging two sound buffers playing in a loop and would like to have fast solution taking less memory.

Edited: I have a solution that looks long:

def sumWithRoll(buffer, indata, idx):
    buffLen = len(buffer)
    dataLen = len(indata)
    if dataLen > buffLen:
        indata = indata[0:buffLen]
        dataLen = buffLen
    idx = idx % buffLen
    idx2 = (idx + dataLen) % buffLen
    if idx2 <= idx:
        idx3 = buffLen - idx
        buffer[idx:buffLen] += indata[0:idx3]                   
        buffer[0:idx2] += indata[idx3:buffLen]
    else:
        buffer[idx:idx2] += indata[:]

I hope there is Pythonic one or two line solution

Henry Ecker · Answer 1 · 2021-05-21T16:25:23.303

1

Try np.roll:

import numpy as np


def sum_with_roll(a, b, offset=0):
    e = np.zeros(a.shape)
    e[tuple(map(slice, b.shape))] = b
    return a + np.roll(e, offset)


a = np.array([1, 1, 1, 1, 1])
b = np.array([5, 6])
print(sum_with_roll(a, b, offset=1))

a = np.array([1, 1, 1, 1, 1])
b = np.array([5, 6, 7, 8])
print(sum_with_roll(a, b, offset=3))

Output:

[1. 6. 7. 1. 1.]
[8. 9. 1. 6. 7.]

For list output:

def sum_with_roll(a, b, offset=0):
    e = np.zeros(a.shape)
    e[tuple(map(slice, b.shape))] = b
    return (a + np.roll(e, offset)).tolist()

[1.0, 6.0, 7.0, 1.0, 1.0]
[8.0, 9.0, 1.0, 6.0, 7.0]

For int type output:

def sum_with_roll(a, b, offset=0):
    e = np.zeros(a.shape)
    e[tuple(map(slice, b.shape))] = b
    return (a + np.roll(e, offset)).astype(int)

[1 6 7 1 1]
[8 9 1 6 7]

edited May 21 '21 at 16:25

answered May 21 '21 at 16:16

Henry Ecker

34,399
18
41
57

Looks concise but was slow when I tested it. Found out "roll" is about 50 times slower than alternatives. This was shown here: https://stackoverflow.com/questions/30399534/shift-elements-in-a-numpy-array – May 21 '21 at 21:16
Nice good to know! I will say, that none of those shift programs do what roll does. – Henry Ecker May 21 '21 at 21:18

Kevin · Accepted Answer · 2021-05-21T19:57:46.403

0

The rolling can be simulated with the simple modulo operator.

Given some indices of where you want to add a and b together, it is essentially just doing a[indices] += b.

a = np.array([1, 1, 1, 1, 1])
b = np.array([5, 6])
offset = 1

indices = np.arange(offset, len(b)+offset) % len(a)
a[indices] += b
# np.add.at(a, indices, b) - if you want to do the operation unbuffered

Output in a:

array([1, 6, 7, 1, 1])

a = np.array([1, 1, 1, 1, 1])
b = np.array([5, 6, 7, 8])
offset = 3

indices = np.arange(offset, len(b)+offset) % len(a)
a[indices] += b

Output in a:

array([8, 9, 1, 6, 7])

You could also use np.bincount for this:

a = np.array([1, 1, 1, 1, 1])
b = np.array([5, 6, 7, 8])
offset = 3

indices = np.arange(offset, len(b)+offset) % len(a)
np.bincount(indices, b)+a

Output:

array([8., 9., 1., 6., 7.])

Unfortunately, it is not possible to specify a result buffer for np.bincount, which would make +a operation obsolete.

EDIT

If you are interested in making it fast I would suggest dividing the in-place add into 2 slices:

a = np.array([1, 1, 1, 1, 1])
b = np.array([5, 6, 7, 8])
offset = 3

size = len(a) - offset
s1 = slice(offset, offset + size)
s2 = slice(None,size)

size = len(b) - size
s3 = slice(None, size)
s4 = slice(size, None)

a[s1] += b[s2]
a[s3] += b[s4]

Output in a:

array([8, 9, 1, 6, 7])

edited May 21 '21 at 19:57

answered May 21 '21 at 17:03

Kevin

3,096
2
8
37

I like your concise solution. Will that modulo operation repeated many times slow it down ? len(b) may be close to million as it is sound samples array. indices = np.arange(offset, len(b)+offset) % len(a) – May 21 '21 at 18:45
It's vectorized, so it should be quite fast in isolation - but you should time it to make sure. The ```np.roll``` method reconstructs the array from slices, so it might be better in terms of the number of reads/writes to memory https://github.com/numpy/numpy/blob/v1.20.0/numpy/core/numeric.py#L1146-L1244 – Kevin May 21 '21 at 19:42
@slmnv5 I made an edit to my post which might give you an entry point of how to optimize the operation. – Kevin May 21 '21 at 19:58
Copy pasted this works, but if parameters change e.g. offset = 2 I got error: a[s3] += b[s4] ValueError: non-broadcastable output operand with shape (1,) doesn't match the broadcast shape (3,) – May 21 '21 at 21:20
I might have made an error somewhere. But the idea is to slice the array into two consecutive parts. The first part are all of the values in b from some offset up until it goes out of bounds and is supposed to "wrap around". The other part are all of the remaining values from the start of b up until the offset. – Kevin May 21 '21 at 23:42

python, summing numpy arrays of same dimensions, different shapes with offset and rolling over the end

2 Answers2