Shift and zero-pad sublists, causes operad/broadcast runtime error that I do not understand

Question

I am trying to do something simple, however I do not understand the resulting error. I have tried googling operands and broadcasting (python says thats what Im doing and Im doing it wrong), but that did not help me. I couldnt find an answer on SO either, but maybe Im using the wrong search terms.

I have a list of 64 sublists which have 64 entries. I want to create a new list of which each sublist has its entries moved to one side ( forward or backward, doesnt matter), then end clipped and the begin padded with zeros so it still has 64 entries. I thought I knew how to do this, but my solution does not work and I do not understand the error. It says I am broadcasting and that Im doing it wrong. However I just made some listcomprehensions (which are just for loops right?) and stuck them together.

Goal:

1 2 3 4 5           1 2 3 4 5
1 2 3 4 5           0 1 2 3 4
1 2 3 4 5  becomes  0 0 1 2 3
1 2 3 4 5           0 0 0 1 2
1 2 3 4 5           0 0 0 0 1

My try:

result = [ [[0 for hh in range(ii)]+originallist[0][jj][0+ii:] for ii in range(64)] for jj in range(64)]

(the extra [0] behind originallist is there because the lists I describe above are actually sublists themselves of one motherlist, but in this example I only look at one such list)

Results in:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-51-5261c3ba129a> in <module>()
----> 1 result = [ [[0 for hh in range(ii)]+originallist[0][jj][0+ii:] for ii in range(64)] for jj in range(64)]

<ipython-input-51-5261c3ba129a> in <listcomp>(.0)
----> 1 result = [ [[0 for hh in range(ii)]+originallist[0][jj][0+ii:] for ii in range(64)] for jj in range(64)]

<ipython-input-51-5261c3ba129a> in <listcomp>(.0)
----> 1 result = [ [[0 for hh in range(ii)]+originallist[0][jj][0+ii:] for ii in range(64)] for jj in range(64)]

ValueError: operands could not be broadcast together with shapes (0) (64)

Related; [Numpy Operands could not be broadcast together with shape](http://stackoverflow.com/q/11856493) — Martijn Pieters, Oct 17 '13 at 10:29
Probably, I would not do a list comprehension for this, since I find it a little bit complicated to understand. I think creating a new variable of the same shape as originallist and then doing the loops would be more clear, although probably there is a more clever way around the problem. — Jblasco, Oct 17 '13 at 10:43

score 1 · Answer 1 · answered Oct 17 '13 at 11:32

the following should do what you need?

l1 = [[1,2,3],[4,5,6],[7,8,9]]
l2 = []
for i,l in enumerate(l1):
    t=[]

    for j in range(len(l)):
        if(j<i):
            t.append(0)
        else:
            t.append(l[j-i])
    l2.append(t)
print l2
# gives: [[1, 2, 3], [0, 4, 5], [0, 0, 7]]

what you wish to do is write this stuff as a oneliner? Personally, i prefer it if it looks like this, since it shows how it works, while oneliners may be clear to you at the moment you program it, but if you look back on them a month later, you have to figure out again how it works so that it does what you want, while if you split it into multiple lines, it is much more self-explanatory

Yes thats what I want to do. Thanks for your answer. If no answer comes that answers all my questions I will accept yours. I was also hoping to understand what went wrong. I dont necessarily want a oneliner, its just that comprehensions are so intuitive. I just say I want a list and write in it what I want to have in it and thats it. No need to think about anything complicated or make any extra variables. — Leo, Oct 17 '13 at 11:46

Daniel · Accepted Answer · 2013-10-17T13:08:21.073

An example without looping

>>> a = np.arange(1,6)
>>> a
array([1, 2, 3, 4, 5])

We now have all of the values and we need to tile. Probably the best way is to use np.tile, but we can use broadcasting here also for a demonstration.

>>> b = np.ones(5,dtype=np.int)
>>> c = a*b[:,None]    #Broadcasting the two together
>>> c
array([[1, 2, 3, 4, 5],
       [1, 2, 3, 4, 5],
       [1, 2, 3, 4, 5],
       [1, 2, 3, 4, 5],
       [1, 2, 3, 4, 5]])

Now that we have your original array we need to manipulate it to return your desired result.

>>> c = c - np.arange(5)[:,None]
>>> c[c<0]=0
>>> c
array([[1, 2, 3, 4, 5],
       [0, 1, 2, 3, 4],
       [0, 0, 1, 2, 3],
       [0, 0, 0, 1, 2],
       [0, 0, 0, 0, 1]])

This is admittedly a limited example. If you can post an example of originallist this can be improved.

Examining your code it looks like originallist is indexing a numpy array. The reason you are getting this error is the + operator will not append a list and a numpy array together and instead tries to broadcast them against each other. So what you are doing is this:

>>> range(0)
[]
>>> [] + np.arange(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (0) (3)

I think you want to do this:

>>> [] + np.arange(3).tolist()
[0, 1, 2]

Ah so thats what is going wrong! I can use `tolist()` in my code and it will work I guess. — Leo, Oct 17 '13 at 15:52

score 1 · Answer 3 · answered Oct 17 '13 at 16:29

1

Just another solution, a quick one-liner:

>>> l1 = [[1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5]]
>>> [([0]*pos)+l[:len(l1)-pos] for pos, l in enumerate(l1)]
[[1, 2, 3, 4, 5], [0, 1, 2, 3, 4], [0, 0, 1, 2, 3], [0, 0, 0, 1, 2], [0, 0, 0, 0, 1]]

answered Oct 17 '13 at 16:29

barmaley

1,027
2
11
12

Nice one, I dont know how to use `for a, b in c` with that comma yet. Will look into `enumerate` as well, it lets you skip a layer there. How I did it after Ophion showed me what I did wrong was: `result = [ [0 for hh in range(jj)]+np.ndarray.tolist(originallist[0][jj][0+jj:]) for jj in range(64)]` Your solution is shorter and easier to read though. Nice! – Leo Oct 17 '13 at 19:47

Shift and zero-pad sublists, causes operad/broadcast runtime error that I do not understand

3 Answers3