Splitting large list into chunks using modulos

Question

I'm wondering why the simple code below doesn't work:

my_list = range(0, 6)

num_proc = 3

chunks = [[]] * num_proc

i = 0
for item in my_list:
    idx = i % num_proc
    i += 1
    chunks[idx].append(item)

print(f"Chunked items = {chunks}")

It outputs:

Chunked items = [[0, 1, 2, 3, 4, 5], [0, 1, 2, 3, 4, 5], [0, 1, 2, 3, 4, 5]]

The output I expect is

Chunked items = [[0, 3], [1, 4], [2, 5]]

Seems that the problem lies with statement

 chunks[idx].append(item)

Yet, I'm at a loss on how to fix that

The reason is explained here: [List of lists changes reflected across sublists unexpectedly](https://stackoverflow.com/questions/240178/list-of-lists-changes-reflected-across-sublists-unexpectedly) — mkrieger1, May 19 '22 at 09:53

Darkstar · Answer 1 · 2022-05-19T10:24:34.743

0

This is happening because [[]] * num_proc creates a reference of the same list num_proc times so any change to one them updates all of them.

So instead of:

chunks = [[]] * num_proc

Do:

chunks = [[] for _ in range(num_proc)]

edited May 19 '22 at 10:24

answered May 19 '22 at 10:01

Darkstar

11
1
3

score 0 · Accepted Answer · answered May 19 '22 at 10:01

The reason is because when you create your chunkslist you create 3 sub-list with the same reference. Therefore when you update it in your code the three sub-lists are updated together.

This line should fix it, it implements chunks in a different way :

chunks = [[] for i in range(num_proc)]

Splitting large list into chunks using modulos

2 Answers2