0

I have two data frames with columns of interest 'ParseCom', which is the left index of this fuzzy join, and 'REF' which should be a substring of 'ParseCom' during a join.

This is iterating over the Dataframe, which is not recommended. How can I implement a fuzzy join in Dask where I am joining on substrings?

for i, com in enumerate(defects['ParseCom']):
    for j, sub in enumerate(repair_matrix['REF']):
        if sub in com:
            print(i,j, com)

enter image description here

Isaacnfairplay
  • 217
  • 2
  • 18

1 Answers1

0

Modifying the approach shown Merge pandas on string contains

Using pandas:

def inmerge(sub,sup, sub_on, sup_on, sub_index, sup_index):  

    sub_par = sub.rename(columns = {sub_on: 'on', sub_index: 'common'})
    sup_par = sup.rename(columns = {sup_on: 'on', sup_index: 'common'})
    print(sub_par.columns.tolist())
    print(sup_par.columns.tolist())
    rhs = (sub_par.on
          .apply(lambda x: sup_par[sup_par.on.str.find(x).ge(0)].common)
          .bfill(axis=1)
          .iloc[:, 0])
    rel =  (pd.concat([sub_par.common, rhs], axis=1, ignore_index=True).rename(columns={0: sub_index, 1: sup_index}))[[sub_index,sup_index]]
    print(rel.columns.tolist())
    print(sub_index,sup_index)
    sub = sub.merge(rel, on = sub_index)
    return sub.merge(sup, on = sup_index)

This has limitations, such as requiring pandas and speed, but it does work faster than a for loop.

Isaacnfairplay
  • 217
  • 2
  • 18