Join two tables with a range criteria using Python Pandas

Question

I have a similar problem to this simplified version:

The experiment result is saved in the excel sheet, and I processed the data using Python Pandas and converted them to DataFrames.

Two tables given below: Table_Race save in DataFrame race Table_standard save in DataFrame std

>>> data = [["Gold+",1,30,35],["Silver+",1,25,30],["Bronze+",1,20,25],["Gold",2,20,25],["Silver",2,15,20],["Bronze",2,10,15]]
>>> std = pd.DataFrame(data,columns=['Title','League','Start','End'])
>>> std
     Title  League  Start  End
0    Gold+       1     30   35
1  Silver+       1     25   30
2  Bronze+       1     20   25
3     Gold       2     20   25
4   Silver       2     15   20
5   Bronze       2     10   15
>>> data = [["John",1,26],["Ryan",1,33],["Mike",1,9],["Jo",2,15],["Riko",2,21],["Kiven",2,13]]
>>> race = pd.DataFrame(data,columns=['Name','League','Distance'])
>>> race
    Name  League  Distance
0   John       1        26
1   Ryan       1        33
2   Mike       1         9
3     Jo       2        21
4   Riko       2        15
5  Kiven       2        13
>>>

I would like to check the distance for each player and get their title according to the standards:

    Title <= distance in [start, end) and need to match league

For example: Jo from league 2 and has distance 15 which is in between [15,20). Note that it's not [10,15), hence he get title 'Silver'

The expected result as follows:

    Name    League  Distance    Title
    John    1       26          Silver+
    Ryan    1       33          Gold+
    Mike    1       9           N/A
    Jo      2       21          Gold
    Riko    2       15          Silver
    Kiven   2       13          Bronze

I can achieved this using two loops which basically get each distance from Table_race and search for (l, d) from each row of race's (League, Distance)

Looking for condition:

    l == League && d >= Start && d < End

But this method is O(N^2) which is too slow, as my data can easily go over 100,000 which takes hours to finish.

Any better solutions?

Above it says in data `Jo 15` and `Riko 21` and in result `Jo 21` and `Riko 15`. — Shashank Singh, Apr 26 '18 at 05:09
thanks for pointing that out, I must made a mistake copy paste, will correct it — ObigOne, Apr 26 '18 at 05:17

Shashank Singh · Accepted Answer · 2018-04-26T05:31:35.020

Still working on solution but here is something to start :

>>> data = [["Gold+",1,30,35],["Silver+",1,25,30],["Bronze+",1,20,25],["Gold",2,20,25],["Silver",2,15,20],["Bronze",2,10,15]]
>>> std = pd.DataFrame(data,columns=['Title','League','Start','End'])
>>> std
     Title  League  Start  End
0    Gold+       1     30   35
1  Silver+       1     25   30
2  Bronze+       1     20   25
3     Gold       2     20   25
4   Silver       2     15   20
5   Bronze       2     10   15

>>> data = [["John",1,26],["Ryan",1,33],["Mike",1,9],["Jo",2,21],["Riko",2,15],["Kiven",2,13]]
>>> race = pd.DataFrame(data,columns=['Name','League','Distance'])
>>> race
    Name  League  Distance
0   John       1        26
1   Ryan       1        33
2   Mike       1         9
3     Jo       2        21
4   Riko       2        15
5  Kiven       2        13
>>> result=pd.merge(race,std,on='League')
>>> result = result[(result.Distance >= result.Start)&(result.Distance < result.End)][["Name","League","Distance","Title"]]
>>> result
     Name  League  Distance    Title
1    John       1        26  Silver+
3    Ryan       1        33    Gold+
9      Jo       2        21     Gold
13   Riko       2        15   Silver
17  Kiven       2        13   Bronze

Merge and Multiple conditions links for their tutorials and drawbacks.

I think it should be `result.Distance >= result.Start`, and this is my answer too. — Lambda, Apr 26 '18 at 05:27

Join two tables with a range criteria using Python Pandas

1 Answers1