0

I have several large lists. I would like to combine them into a one-denominational list. For example,

small_lists = [[{'value1':1}]*100000,[{'value2':2}]*100000,[{'value3':3}]*100000]
combined_list = []
for small_list in small_lists:
    combined_list.extend(small_list)

Is there a faster way than above?

numpy is suggested in several answers but it seems significant slower for me. Am I doing anything wrong?

import time
import numpy as np

small_lists = [[{'value1':1}]*10000000,[{'value2':2}]*10000000,[{'value3':3}]*10000000]

start = time.time()
np_list = np.array(small_lists).flatten()
print("{} sec".format(time.time() - start))
print(len(np_list))

start = time.time()
combined_list = []
for small_list in small_lists:
    combined_list.extend(small_list)
print("{} sec".format(time.time() - start))
print(len(combined_list))

from functools import reduce
start = time.time()
reduce_list = reduce(lambda x, y: x+y, small_lists)
print("{} sec".format(time.time() - start))
print(len(reduce_list))

The output is 2.01335906982 sec for numpy, 0.113998889923 sec for extend, 0.299326896667 sec for reduce. extend is by far the fastest.

abisko
  • 663
  • 8
  • 21
  • 1
    Look at [this answer](https://stackoverflow.com/a/45323085/4996248) to a related (almost duplicate) question for some timings. – John Coleman May 27 '20 at 16:25
  • Also [this answer](https://stackoverflow.com/questions/17044508/what-is-the-fastest-way-to-merge-two-lists-in-python) – DarrylG May 27 '20 at 16:31
  • `numpy` isn't really optimized for things like arrays of dictionaries. I wouldn't be surprised if `numpy` was better for lists of ints but not lists of dictionaries. – John Coleman May 27 '20 at 17:31

2 Answers2

0

Use numpy - for extremely large arrays this can be 10-100 times faster:

import numpy as np

np_array = np.array(small_list)
flat = np_array.flatten()
Aziz Sonawalla
  • 2,482
  • 1
  • 5
  • 6
  • Am I doing anything wrong? numpy seems a lot slower for me. Sorry I can't insert code here. Will add a new comment onto my original question – abisko May 27 '20 at 16:37
0
from functools import reduce

combined_list = reduce(lambda x, y: x + y, small_lists)

https://docs.python.org/3.7/library/functools.html#functools.reduce

captnsupremo
  • 689
  • 5
  • 11