Count word frequency in tokenized word - with else if logic

Question

I am trying to count the word frequency is a list in a dataframe.

data = {'H':[['the', 'brown', 'fox'], ['the', 'weather', 'is'],['she', 'sells', 'sea']], 'marks':['a', 'b', 'c']} 
df = pd.DataFrame(data)

I want to separate the word counts based when marks is a,b,c. I know I can make x3 separate dataframes but I am looking for a cleaner code output

freq = {}
  def count_freq(word):
     for w in word:
         if w in list(freq.keys()):
            freq[w] += 1
      else:
        freq[w] = 1

df.H.apply(count_freq)

then I tried this but I am messing up

df['marks'] = z.apply(lambda row: 0 if row['marks'] in ("a")
             else if row['marks'] in ("b")
             else row['marks'] in ("c")

Edit: expected outcome

            Frequency-a   Frequency-b    Frequency-c    
the         1              1
quick       1
brown       1
fox         1
she                                       1
sells                                     1
sea                                       1
weather                    1
is                         1

Once again, could you please indicate what the expected output would look like? — cs95, Apr 08 '19 at 19:06

cs95 · Answer 1 · 2019-04-08T19:19:17.700

2

You can use get_dummies and transpose the result:

df['H'].str.join(',').str.get_dummies(sep=',').set_index(df['marks']).T

marks    a  b  c
brown    1  0  0
fox      1  0  0
is       0  1  0
sea      0  0  1
sells    0  0  1
she      0  0  1
the      1  1  0
weather  0  1  0

edited Apr 08 '19 at 19:19

answered Apr 08 '19 at 19:13

cs95

379,657
97
704
746

score 2 · Accepted Answer · answered Apr 08 '19 at 19:15

From sklearn MultiLabelBinarizer

from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()
print (pd.DataFrame(mlb.fit_transform(df['H'].values),columns=mlb.classes_, index=df.marks).T)
marks    a  b  c
brown    1  0  0
fox      1  0  0
is       0  1  0
sea      0  0  1
sells    0  0  1
she      0  0  1
the      1  1  0
weather  0  1  0

score 2 · Answer 3 · answered Apr 08 '19 at 19:16

2

You can unnest with crosstab:

u = unnesting(df, 'H')
pd.crosstab(u.H, u.marks)

marks    a  b  c
H
brown    1  0  0
fox      1  0  0
is       0  1  0
sea      0  0  1
sells    0  0  1
she      0  0  1
the      1  1  0
weather  0  1  0

answered Apr 08 '19 at 19:16

user3483203

50,081
9
65
94

which function in python has an unnest feature? – OptimusPrime Apr 08 '19 at 19:25
If you follow the link, the answer by Wen contains the `unnest` function. It was part of a canonical answer he wrote, and is a handy tool to use. – user3483203 Apr 08 '19 at 19:26

Count word frequency in tokenized word - with else if logic

3 Answers3