Python Find the words in the set have word pairs on dataframe

Question

I have set of words

{'DNA',
 'pada',
 'yang',...}

I am trying to find whether the words in the set have word pairs on the following bigram dataframe

                bigramf    freq
0           (DNA, yang)      15
1          (DNA, dalam)       6
2            (DNA, ini)       5
3       (DNA, memiliki)       4
4         (DNA, unting)       4
5           (pada, DNA)       4
6        (pada, urutan)       3
7     (yang, diperoleh)       3
8         (yang, lebih)       3
9      (pada, sejumlah)       2
10          (pada, RNA)       2
11          (pada, satu)      2
12       (yang, berbeda)      2     
13          (yang, sama)      2
14       (yang, tumpang)      2

If so, then the expected output will be like this:

yang [('lebih', 3), ('diperoleh', 3), ('berbeda', 2), ('tumpang', 2), ('sama', 2)]

DNA [('yang', 15), ('dalam', 6), ('ini', 5), ('memiliki', 4), ('unting', 4)]

pada [('DNA', 4), ('urutan', 3), ('sejumlah', 2), ('RNA', 2), ('satu', 2)]

How do I find?. Can anyone can help? Thanks. Any help is much appreciated.

jezrael · Answer 1 · 2020-02-20T06:28:44.607

First convert column with tuples to new 2 columns (dont use apply(pd.Series), because slow), then filter matched values by Series.isin in boolean indexing and convert in GroupBy.apply values to list of tuples:

s = {'DNA',
 'pada',
 'yang'}

df[['s', 'v']] = pd.DataFrame(df['bigramf'].tolist(), index=df.index)

s = df[df['s'].isin(s)].groupby('s')['v','freq'].apply(lambda x: list(map(tuple, x.values)))
print (s)
s
DNA     [(yang, 15), (dalam, 6), (ini, 5), (memiliki, ...
pada    [(DNA, 4), (urutan, 3), (sejumlah, 2), (RNA, 2...
yang    [(diperoleh, 3), (lebih, 3), (berbeda, 2), (sa...
dtype: object

If need dictionary add Series.to_dict:

d = s.to_dict()
print (d)
{'DNA': [('yang', 15), ('dalam', 6), ('ini', 5), ('memiliki', 4), ('unting', 4)], 
 'pada': [('DNA', 4), ('urutan', 3), ('sejumlah', 2), ('RNA', 2), ('satu', 2)], 
 'yang': [('diperoleh', 3), ('lebih', 3), ('berbeda', 2), ('sama', 2), ('tumpang', 2)]}

Another solution with collections.defaultdict:

from collections import defaultdict

d = defaultdict(list)
for (s1, v1), f1 in df.to_numpy():
    if s1 in s:
        d[s1].append((v1, f1))

d = dict(d)
print (d)
{'DNA': [('yang', 15), ('dalam', 6), ('ini', 5), ('memiliki', 4), ('unting', 4)], 
 'pada': [('DNA', 4), ('urutan', 3), ('sejumlah', 2), ('RNA', 2), ('satu', 2)], 
 'yang': [('diperoleh', 3), ('lebih', 3), ('berbeda', 2), ('sama', 2), ('tumpang', 2)]}

@ohai - Sure, it test if first values of bigrams are in sets `s = {'DNA', 'pada', 'yang'}` — jezrael, Feb 20 '20 at 08:09
@ohai - In second solution is used `if s1 in s:` with same effect — jezrael, Feb 20 '20 at 08:09

score 0 · Answer 2 · answered Feb 20 '20 at 06:19

0

What I did is for matching words

I looped through the data frame and took each element and passed it to a def like this

def match_words(actual_word, word):
    return set(actual_word.split()).intersection(word.split())

answered Feb 20 '20 at 06:19

Krk Rama Krishna

117
1
4

Python Find the words in the set have word pairs on dataframe

2 Answers2