memory error while extending lists

Question

I have such a list:

mylist=[[[1,2,3,...,200],[1,2,3,...,200],[1,2,3,...,200],[1,2,3,...,40]],...]

I want to merge the first three lists in each item in mylist and have something like this:

final_list=[[[1,2,3,...,600],[1,2,3,...,40]],...]

I have written this code to do that:

def extend(mylist):
    a=mylist[0]
    b=mylist[1]
    c=mylist[2]
    a.extend(b)
    a.extend(c)
    return a

final_list=[]
for i in mylist:
    final_list.append([extend(i),i[3]])

it works properly when the lists are small, but when I want to merge three 200-array lists into a 600-array list, I face "MemorryError":

    a.extend(c)
MemoryError

How can this problem be solved?

You should try to use generators with yield, see https://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do — Igl3, Aug 11 '17 at 11:34

score 1 · Accepted Answer · answered Aug 11 '17 at 13:10

First of all, I'm not sure this is gonna work.

I created your list using a generator to avoid having all your lists in memory. And using itertools.chain to improve efficiency while concatenating your sublists.

import itertools

mylist = [[[1, 2], [3, 4], [5, 6], [7, 8]]]

def contact_elements(huge_list):
  for i in huge_list:
    yield [list(itertools.chain(i[0], i[1], i[2])), i[3]]

final_list = [sub for sub in contact_elements(mylist)]
print(final_list) # [[[1, 2, 3, 4, 5, 6], [7, 8]]]

memory error while extending lists

1 Answers1