How to collapse a data frame column of lists into one list for NLP

Question

I am trying to do some topic analysis and I need to collapse a column in dataframe made up of lists, into one list of words.

So here's an approximation of what my data looks like:

import pandas as pd
d = {'Case': ["[wait, information, employer]","[case, assign, priority, level, 2, transmit]" ]}
df2 = pd.DataFrame(data=d)

I would like to get one list like list = ['wait', 'information', 'case','assign','priority', 'level']

The values of "Case" column are strings or lists of strings? — Andrej Kesely, Jun 02 '21 at 14:39
you wanna do something like this? https://stackoverflow.com/questions/34238391/convert-one-dataframe-row-to-flat-list — Matt Camp, Jun 02 '21 at 14:40

score 1 · Accepted Answer · answered Jun 02 '21 at 14:43

1

IIUIC you could do something like this to get a list from your column

import nltk
token = []
token = token.append(df['Case'].apply(lambda x: nltk.word_tokenize(x)))

answered Jun 02 '21 at 14:43

slb20

score 1 · Answer 2 · answered Jun 02 '21 at 14:44

1

df2["CaseList"] = df2["Case"].apply(lambda x: x.replace("]","").replace("[","").split(","))

Does that work for you?

answered Jun 02 '21 at 14:44

Matt Camp

2 Answers2