scala fast range lookup on 2 columns

Asked Jul 02 '19 at 18:51

Active Jul 04 '19 at 06:24

Viewed 199 times

I have a spark dataframe that I am broadcasting as Array[Array[String]]. My requirement is to do a range lookup on 2 columns.

Right now I have something like ->

val cols = data.filter(_(0).toLong <= ip).filter(_(1).toLong >= ip).take(1) match {
    case Array(t) => t
    case _ => Array()
  }

The following data file is stored as Array[Array[String]] (except for the header row that I have shown below only as reference.) and passed to the filter function shown above.

sample data file ->

startIPInt | endIPInt  | lat               | lon
676211200  | 676211455 | 37.33053          | -121.83823
16777216   | 16777342  | -34.9210644736842 | 138.598709868421
17081712   | 17081712  | 0                 | 0

sample value to search ->

ip = 676211325

based on the range of the startIPInt and endIPInt values, I want the rest of the mapping rows.

This lookup takes 1-2 sec for each, and I am not even sure the 2nd filter condition is getting executed(in debug mode always it only seems to execute the 1st condition). Can someone suggest me a faster and more reliable lookup here?

Thanks!

edited Jul 03 '19 at 17:34

asked Jul 02 '19 at 18:51

user3868051

1,147
2
22
43

1

Can you add sample input DataFrames and expected output please? – C.S.Reddy Gadipally Jul 02 '19 at 18:58
@C.S.ReddyGadipally added sample – user3868051 Jul 02 '19 at 22:42
I would love to help you but your question is still unclear to me. It helps if you add all your inputs ( samples) and expected output. Please see this thread for more information https://stackoverflow.com/questions/48427185/how-to-make-good-reproducible-apache-spark-examples – C.S.Reddy Gadipally Jul 03 '19 at 15:20
I think is unclear what are you asking. Can you provide an example of relation between Array[Array[String]] and sample data? – abiratsis Jul 03 '19 at 16:19
@C.S.ReddyGadipally – user3868051 Jul 03 '19 at 17:38
@AlexandrosBiratsis – user3868051 Jul 03 '19 at 17:38

scala fast range lookup on 2 columns

0 Answers0