Optimize making a flat list from a list of strings each evaluable as a list

Question

For example, how would I optimally merge:

res_str = ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']
for example: ['[{'a': u'中国', 'b': u'美国', 'c': u'日本', 'd': u'德国', 'e': u'法国'},]','[{'a': u'中国', 'b': u'美国', 'c': u'日本', 'd': u'德国', 'e': u'法国'},]',]

into:

[1,2,3,4,5,6,7,8,9,10,11,12]

I used the following code, but is is not fast enough:

[x for j in res_str for x in eval(j)]  spend time 0.65s
list(itertools.chain.from_iterable([eval(i) for i in res_str]))  spend time 0.57s

Is there a better way to write this?

apart from a generator

(x for j in res_str for x in eval(j))

other way

sum([eval(i) for i in res_str],[]) spend time 3.87s

this way:

import ast
import itertools
l = ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']
l = list(itertools.chain(*map(ast.literal_eval, l)))
spend time 0.95s

if use eval

list(itertools.chain(*map(eval, res_str)))
spend  time 0.58s

this way:

eval('+'.join('+'.join(arr)))  spend time 3.5s

this way:

import ast
import numpy as np
res_str = ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']
print(list(np.array([ast.literal_eval(i) for i in res_str]).flatten()))
spend time 1s

if use eval list(np.array([eval(i) for i in res_str]).flatten()) spend time 0.58s

Possible duplicate of [Making a flat list out of list of lists in Python](https://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-lists-in-python) — Mihai Alexandru-Ionut, Jun 22 '18 at 07:01
@MihaiAlexandru-Ionut That other question doesn't cover the key point here: how to deal most efficiently with conversion from string. So **it is not a duplicate**. The OP already shows a solution to that other question, that's not what the OP is asking about. — Jean-François Corbett, Jun 22 '18 at 08:36
@xin.chen You say your method isn't fast enough... and all the comments you've posted on answers given so far say that they're not fast enough either. What kind of performance improvement are you hoping for? Perhaps you could show us your time measurement of the various methods? — Jean-François Corbett, Jun 22 '18 at 08:42

score 1 · Answer 1 · answered Jun 22 '18 at 07:03

1

Use ast & itertools

Ex:

import ast
import itertools
l = ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']
l = list(itertools.chain(*map(ast.literal_eval, l)))
print( l )

Output:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

ast.literal_eval to convert string elements to list objects
itertools.chain to flatten the list.

answered Jun 22 '18 at 07:03

Rakesh

81,458
17
76
113

This method is not fast enough，my list length 22443, speed time 0.95s – xin.chen Jun 22 '18 at 07:39

score 0 · Answer 2 · answered Jun 22 '18 at 07:31

0

If you like to do it without eval / ast.literal_eval

>>> list(itertools.chain(*[map(int, w.strip('[]').split(',')) for w in l]))
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

answered Jun 22 '18 at 07:31

Sunitha

11,777
2
20
23

Anand Tripathi · Answer 3 · 2018-06-22T13:40:04.410

Using List Comprehension

import json
string_list = ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']
output_list = [y for x in string_list for y in json.loads(x)]
print output_list

OUTPUT

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

Here the time taken is only to traverse the list and space complexity is the new list.

import json
string_list = ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']
output_list = []
for str_list in string_list:
    output_list.extend(json.loads(str_list))
print output_list

Output

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

score 0 · Answer 4 · answered Jun 22 '18 at 07:44

Encountered this problem a while ago. Here's how I did it

List = [[1,2,3],[4,5,6],[7,8,9]]
result = []
for x in range(len(List)):
    for y in range(len(List[x])):
        result.append(List[x][y])

print(result)

Result prints [1,2,3,4,5,6,7,8,9]

May not be as efficient as some other answers but it works and is more simple

Laurent H. · Answer 5 · 2018-06-22T07:56:08.277

Here is my one-line stylish solution without using itertools and easily readable:

import ast

myList= ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']

myNewList = [i for sublist in map(ast.literal_eval, myList) for i in sublist]

print(myNewList)
# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

Here is also a second solution that may be faster:

import ast

myList = ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']

myNewList = []

for sublist in myList:
    myNewList += ast.literal_eval(sublist)

print(myNewList)
# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

hygull · Answer 6 · 2018-06-22T08:11:19.523

0

You can try the below simple approach.

>>> arr
['[1,2,3]', '[4,5,6]', '[7,8,9]', '[10,11,12]']
>>>
>>> '+'.join(arr)
'[1,2,3]+[4,5,6]+[7,8,9]+[10,11,12]'
>>>
>>> eval('+'.join(arr))
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
>>>

Another approach using reduce() and lambda.

>>> import json
>>>
>>> arr = ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']
>>>
>>> arr2 = reduce(lambda list1, list2: list1 + '+' + list2, arr)
>>>
>>> arr2
'[1,2,3]+[4,5,6]+[7,8,9]+[10,11,12]'
>>>
>>> eval(arr2)
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

edited Jun 22 '18 at 08:11

answered Jun 22 '18 at 08:05

hygull

8,464
2
43
52

Okay, I've added another way. – hygull Jun 22 '18 at 08:11
same data ,Slower than the previous one – xin.chen Jun 22 '18 at 08:16
Okay, please tell me the execution time. I'll try to find a better way. Thanks. – hygull Jun 22 '18 at 08:19
my list length 22000, ['[{},{}]','[{},{},{}]'] the first way spend time:3.9s the second way spend time:8.8s – xin.chen Jun 22 '18 at 08:44
And what is expected? So that I could try to do in that range. – hygull Jun 22 '18 at 08:53
for example: ['[{'a': u'中国', 'b': u'美国', 'c': u'日本', 'd': u'德国', 'e': u'法国'},]','[{'a': u'中国', 'b': u'美国', 'c': u'日本', 'd': u'德国', 'e': u'法国'},]',] – xin.chen Jun 22 '18 at 09:07
I am asking about time (minimum) that you expect. – hygull Jun 22 '18 at 11:22
if length 22000 i wangt spend 0.1s or less than 0.1s – xin.chen Jun 22 '18 at 11:31

score 0 · Answer 7 · answered Jun 22 '18 at 10:29

ast.literal_eval+numpy.flatten:

import ast
import numpy as np
res_str = ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']
print(list(np.array([ast.literal_eval(i) for i in res_str]).flatten()))

and:

import ast
l = []
res_str = ['[1,2,3]','[4,5,6]','[7,8,9]','[10,11,12]']
for i in res_str:
   l.extend(ast.literal_eval(i))
print(l)

Optimize making a flat list from a list of strings each evaluable as a list

7 Answers7