5

I have a pandas dataframe as:

word_list
['nuclear','election','usa','baseball']
['football','united','thriller']
['marvels','hollywood','spiderman']
....................
....................
....................

I also have multiple number of lists with categories names,something as:-

movies=['spiderman','marvels','thriller']'

sports=['baseball','hockey','football'],

politics=['election','china','usa'] and many others categories.

All I want to match the keywords of pandas column word_list with my categories lists and assign the corresponding lists name in seperate column if keywords gets matched together and if any keywords not gets matched in any of the list then simply put as miscellaneous So, output I'm looking for as:-

word_list                                          matched_list_names
['nuclear','election','usa','baseball']            politics,sports,miscellaneous
['football','united','thriller']                   sports,movies,miscellaneous               
['marvels','spiderman','hockey']                   movies,sports

....................                               .....................
....................                               .....................
....................                               ....................

I successfully get the match keywords as:-

for i in df['word_list']:
    for j in movies:
        if i in j:
           print (i)

but this gives me the list of matched keywords. How I get the list names and add it into pandas column?

Learner
  • 800
  • 1
  • 8
  • 23
  • You are asking multiple questions now. But about the relevance (1/3)*100 is actually 0.3333.... Are you not content with the current answers? – Anton vBR Jul 29 '18 at 19:24
  • @AntonvBR I need to calculate the relevance value also. So, 0.33 can also be used. I have tried alot but my approach doesn't seems to work for me. – Learner Jul 30 '18 at 04:48

2 Answers2

3

You can flatten dictionary of lists first and then lookup by .get with miscellaneous for non matched values, then convert to sets for unique categories and convert to strings by join:

movies=['spiderman','marvels','thriller']
sports=['baseball','hockey','football']
politics=['election','china','usa']
d = {'movies':movies, 'sports':sports, 'politics':politics}
d1 = {k: oldk for oldk, oldv in d.items() for k in oldv}

f = lambda x: ','.join(set([d1.get(y, 'miscellaneous') for y in x]))
df['matched_list_names'] = df['word_list'].apply(f)
print (df)

                                 word_list             matched_list_names
0       [nuclear, election, usa, baseball]  politics,miscellaneous,sports
1             [football, united, thriller]    miscellaneous,sports,movies
2  [marvels, hollywood, spiderman, budget]           miscellaneous,movies

Similar solution with list comprehension:

df['matched_list_names'] = [','.join(set([d1.get(y, 'miscellaneous') for y in x])) 
                            for x in df['word_list']]
jezrael
  • 822,522
  • 95
  • 1,334
  • 1,252
  • Thanks for the comment. Please,check the edited question. – Learner Jul 29 '18 at 14:47
  • I have added a seperate question for my query `https://stackoverflow.com/questions/51589060/add-numeric-values-beside-columns-elements-in-pandas` – Learner Jul 30 '18 at 07:36
  • Need your help. Can you please take a look into this `https://stackoverflow.com/questions/51864822/mapping-of-pandas-column-with-column-of-another-pandas-dataframe/51864988#51864988` – Learner Aug 15 '18 at 19:37
1

First of all, I think you should take advantage of O(1) lookup from sets and dictionaries. That said, I'd set the data as (notice that values are sets):

d = dict(movies={'spiderman','marvels','thriller'},
         sports={'baseball','hockey','football'},
         politics={'election','china','usa'})

Then, you can transform your series using your custom logic

def f(r):
    def m(r_):
        _ = [k for (k, v) in d.items() if r_ in v]
        return _ if _ else ['Misc']
    return {item for z in [m(r_) for r_ in r] for item in z}

df.word_list.transform(f)

0    {Misc, sports, politics}
1      {Misc, sports, movies}
2              {Misc, movies}

For 300000 rows,

%timeit df.word_list.transform(f)
1.1 s ± 22.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

which is not great but doable..

rafaelc
  • 57,686
  • 15
  • 58
  • 82