How to convert list column to a non-nested list?

Question

How to convert a column to a non-nested list while the column elements are list?

For example, the column is like

column
[1, 2, 3]
[1, 2]

I want following at last.

[1,2,3,1,2]

But now with column.tolist(), I will get

[[1,2,3],[1,2]]

EDIT: Thanks for help. My intention is to find the most simple (elegant) and efficient method to do this. Now I use @jezrael method.

from itertools import chain
output = list(chain.from_iterable(df[column])

The simplest method is provided by @piRSquared, but maybe slower.

output = df[column].values.sum()

jezrael · Accepted Answer · 2017-06-26T06:40:07.483

7

You can use numpy.concatenate:

print (np.concatenate(df['column'].values).tolist())
[1, 2, 3, 1, 2]

Or:

from  itertools import chain
print (list(chain.from_iterable(df['column'])))
[1, 2, 3, 1, 2]

Another solution, thanks juanpa.arrivillaga:

print ([item for sublist in df['column'] for item in sublist])
[1, 2, 3, 1, 2]

Timings:

df = pd.DataFrame({'column':[[1,2,3], [1,2]]})
df = pd.concat([df]*10000).reset_index(drop=True)
print (df)

In [77]: %timeit (np.concatenate(df['column'].values).tolist())
10 loops, best of 3: 22.7 ms per loop

In [78]: %timeit (list(chain.from_iterable(df['column'])))
1000 loops, best of 3: 1.44 ms per loop

In [79]: %timeit ([item for sublist in df['column'] for item in sublist])
100 loops, best of 3: 2.31 ms per loop

In [80]: %timeit df.column.sum()
1 loop, best of 3: 1.34 s per loop

edited Jun 26 '17 at 06:40

answered Jun 26 '17 at 05:57

jezrael

822,522
95
1,334
1,252

Thanks? But whether there exists any other `method` could do `concat` and `tolist` at one time? – danche Jun 26 '17 at 05:59
1

@danche unlikely. `pandas` wasn't designed to hold `list`s as values. – juanpa.arrivillaga Jun 26 '17 at 06:00
@juanpa.arrivillaga - Thanks – jezrael Jun 26 '17 at 06:02
@juanpa.arrivillaga Thks! – danche Jun 26 '17 at 06:04
@danche - in pandas is possible working with another values as scalar, but many function does not work. So best is avoiding it. – jezrael Jun 26 '17 at 06:06
@jezrael Thks for the your advice. :-) – danche Jun 26 '17 at 06:08
1

@juanpa.arrivillaga - I prefer your solution long time, but after reading [this timings](https://stackoverflow.com/a/953097/2901002) I start prefer `chain.from_iterable` what is faster. – jezrael Jun 26 '17 at 06:10
1

@jezrael wow, I use both those approaches frequently, never really preferring one or the other. I figured one would be marginally better, and never imagined that the difference would be by a factor of 2. Good to know. – juanpa.arrivillaga Jun 26 '17 at 06:16

piRSquared · Answer 2 · 2017-06-26T06:51:54.277

2

We concatenate lists with the + operator. Because a pandas series uses its' elements underlying + operation when you call pd.Series.sum, we can concatenate a whole column, or series, of lists with.

df.column.sum()

[1, 2, 3, 1, 2]

But if you're looking for performance, you can consider cytoolz.concat

import cytoolz

list(cytoolz.concat(df.column.values.tolist()))

[1, 2, 3, 1, 2]

edited Jun 26 '17 at 06:51

answered Jun 26 '17 at 06:37

piRSquared

285,575
57
475
624

Hmmm, but it is really slow... plus1 – jezrael Jun 26 '17 at 06:40
1

:-) not trying to be fast on this one. But it is the simplest. – piRSquared Jun 26 '17 at 06:43
exactly, it is simpliest. – jezrael Jun 26 '17 at 06:44

score 0 · Answer 3 · answered Jun 26 '17 at 06:14

0

You can use append method of list to do this:

col = {'col': [[1, 2, 3], [1, 2]]}
last = []
last.extend([i for c in col['col'] for i in c])

answered Jun 26 '17 at 06:14

Pooja

1,254
14
17

score 0 · Answer 4 · answered Jun 26 '17 at 06:43

0

Another solution that will work is the list.extend() method.

list = [] for row in column: list.extend(row)

answered Jun 26 '17 at 06:43

Jason Stein

714
3
10

How to convert list column to a non-nested list?

4 Answers4

Linked

Related