Extract values from a column of lists

Question

I have the following data frame, the value of the column sequence is a list:

id      sequence
001    [A, B, C, E, F]
002    [A, C]
003    []
004    [D]

I want to create two new columns called first and second_to_last: first indicating the first element of the list in the sequence column, second_to_last indicating the second to last element of the list in the sequence column. I am expecting the new df to be like:

id      sequence             first    second_to_last
001    [A, B, C, E, F]        A        E
002    [A, C]                 A        A
003    []                     None     None
004    [D]                    D        None

I tried to use the following code:

df['first'] = df['sequence'][0]
df['second_to_last'] = df['sequence'][-2]

But got the following errors:

There was a problem running this cell
ValueError Length of values does not match length of index 
ValueErrorTraceback (most recent call last)
<ipython-input-9-f08abfd1f93c> in <module>()

----> 2 df['first'] = df['sequence'][0]
      3 df['second_to_last'] = df['sequence'][-2]
      4 df

/opt/conda/envs/python2/lib/python2.7/site-packages/pandas/core/frame.pyc in __setitem__(self, key, value)
   2427         else:
   2428             # set column
-> 2429             self._set_item(key, value)
   2430 
   2431     def _setitem_slice(self, key, value):

/opt/conda/envs/python2/lib/python2.7/site-packages/pandas/core/frame.pyc in _set_item(self, key, value)
   2493 
   2494         self._ensure_valid_index(value)
-> 2495         value = self._sanitize_column(key, value)
   2496         NDFrame._set_item(self, key, value)
   2497 

/opt/conda/envs/python2/lib/python2.7/site-packages/pandas/core/frame.pyc in _sanitize_column(self, key, value, broadcast)
   2664 
   2665             # turn me into an ndarray
-> 2666             value = _sanitize_index(value, self.index, copy=False)
   2667             if not isinstance(value, (np.ndarray, Index)):
   2668                 if isinstance(value, list) and len(value) > 0:

/opt/conda/envs/python2/lib/python2.7/site-packages/pandas/core/series.pyc in _sanitize_index(data, index, copy)
   2877 
   2878     if len(data) != len(index):
-> 2879         raise ValueError('Length of values does not match length of ' 'index')
   2880 
   2881     if isinstance(data, PeriodIndex):

ValueError: Length of values does not match length of index

What should be the correct way of extract values for column first and second_to_last? Thanks!

score 3 · Accepted Answer · answered Mar 29 '18 at 03:47

Option 1
You'll need to use the str accessor when dealing with columns of strings/other mutable objects in pandas.

df['first'] = df['sequence'].str[0]
df['second_to_last'] = df['sequence'].str[-2]

df
   id         sequence first second_to_last
0   1  [A, B, C, E, F]     A              E
1   2           [A, C]     A              A
2   3               []   NaN            NaN
3   4              [D]     D            NaN

Option 2
Another option would be defining your own function to retrieve items at the their given index:

def get_value(d, i):
    try:
        return d[i]
    except IndexError:
        return np.nan

Loop over df.sequence:

df['first'] = [get_value(d, 0) for d in df.sequence]
df['second_to_last'] = [get_value(d, -2) for d in df.sequence]

df

   id         sequence first second_to_last
0   1  [A, B, C, E, F]     A              E
1   2           [A, C]     A              A
2   3               []   NaN            NaN
3   4              [D]     D            NaN

I didn't know the `NDFrame.str` works for sequence other than string. — Eastsun, Mar 29 '18 at 07:06

score 2 · Answer 2 · answered Mar 29 '18 at 03:59

This method is not efficient compare with cold's magic str

df['seq'].apply(lambda x : np.nan if not x else x[0])
Out[1328]: 
0      A
1      A
2    NaN
3      D
Name: seq, dtype: object
df['seq'].apply(lambda x : np.nan if not x or len(x)<2 else x[-2])
Out[1329]: 
0      E
1      A
2    NaN
3    NaN
Name: seq, dtype: object

Extract values from a column of lists

2 Answers2

Linked