0

I am having a dataframe with following values:

sentence_id  words                    labels
3822445      ['a', 'b', 'c', '']      ['B-PER', 'I-PER', 'I-PER', 'I-PER']
3822446      ['d', 'e', '']           ['B-PER', 'I-PER', 'I-PER']
3822447      ['f', 'g', 'h']          ['B-PER', 'I-PER', 'I-PER']

Exepcting output as:

sentence_id  words    labels    
3822445       'a'     'B-PER'
3822445       'b'     'I-PER'
3822445       'c'     'I-PER'
3822445       ''      'I-PER'
3822446       'd'     'B-PER'
3822446       'e'     'I-PER'
3822446       ''      'I-PER'
3822447       'f'     'B-PER'
3822447       'g'     'I-PER'
3822447       'h'     'I-PER'

I have tried:

dataframe.set_index(['sentence_id']).apply(pd.Series.explode).reset_index()

but giving same output as input. Don't know what's going wrong.

Shyam
  • 357
  • 1
  • 9
  • 1
    Looks like: https://stackoverflow.com/questions/63583059/split-multiple-columns-of-lists-into-separate-rows. Just remove the `.query` part from my solution there since you want to keep the empty strings. – ALollz Feb 22 '21 at 17:35

2 Answers2

3

If you want a simple one-liner you can use explode with pandas>=0.25.0

df.explode('words').assign(labels=df['labels'].explode())
Ric S
  • 9,073
  • 3
  • 25
  • 51
  • Actually it was a silly mistake, when I was reading csv it was taking list of words as list in string – Shyam Feb 23 '21 at 05:40
1

Update for pandas 1.3.0

pandas.DataFrame.explode now accepts a list of column headers

df.explode(['words','labels'], ignore_index=True)

Output:

   sentence_id words labels
0      3822445     a  B-PER
1      3822445     b  I-PER
2      3822445     c  I-PER
3      3822445        I-PER
4      3822446     d  B-PER
5      3822446     e  I-PER
6      3822446        I-PER
7      3822447     f  B-PER
8      3822447     g  I-PER
9      3822447     h  I-PER

This works fine with me. What are your unexpected results?

df  = pd.DataFrame({'sentence_id':[3822445, 3822446, 3822447],
                    'words':[['a', 'b', 'c', ''],
                            ['d', 'e', ''],
                            ['f', 'g','h']],
                   'labels':[['B-PER', 'I-PER', 'I-PER', 'I-PER'],
                            ['B-PER','I-PER', 'I-PER'],
                            ['B-PER', 'I-PER','I-PER']]})

df.set_index('sentence_id').apply(pd.Series.explode).reset_index()

Output:

   sentence_id words labels
0      3822445     a  B-PER
1      3822445     b  I-PER
2      3822445     c  I-PER
3      3822445        I-PER
4      3822446     d  B-PER
5      3822446     e  I-PER
6      3822446        I-PER
7      3822447     f  B-PER
8      3822447     g  I-PER
9      3822447     h  I-PER
Scott Boston
  • 147,308
  • 15
  • 139
  • 187
  • Actually it was a silly mistake, when I was reading csv it was taking list of words as list in string. – Shyam Feb 23 '21 at 05:40