0

I have following code :-

 var ArrayStop = new ArrayList[imeistoppage]()
 val listings = rddStopCalculate.zipWithIndex().map(p => {
        if (p._2 == 0) {
          imei = p._1.imei
          lat = p._1.latitude
          long = p._1.longitude
          gpsdt = p._1.gpsdt
        } else if (p._2 > 0 && p._2 != lastindex) {
          if (p._1.imei.equals(imei) && p._1.latitude == lat && p._1.longitude == long) {
            flag += 1
            newgpsdt = p._1.gpsdt
          } else {
            if (flag > 0) {
              timeDiff = newgpsdt.getTime() - gpsdt.getTime()
              if (timeDiff > 60000) {
                ArrayStop.add(imeistoppage(p._1.imei, lat, long, timeDiff))
              }
              flag = 0
            }
            imei = p._1.imei
            lat = p._1.latitude
            long = p._1.longitude
            gpsdt = p._1.gpsdt
          }
        } else {
          if (p._1.imei.equals(imei) && p._1.latitude == lat && p._1.longitude == long) {
            flag += 1
            newgpsdt = p._1.gpsdt
          }
          if (flag > 0) {
            timeDiff = newgpsdt.getTime() - gpsdt.getTime()
            if (timeDiff > 60000) {
              ArrayStop.add(imeistoppage(p._1.imei, lat, long, timeDiff))
            }
            flag = 0
          }
        }
        ArrayStop
      }).collect()

      val returnList = listings(listings.length - 1)
      val tempCollection = returnList.asScala
      val tempRDD = sc.parallelize(tempCollection)
      tempRDD.saveToCassandra("db", "table", SomeColumns("imei", "lat", "long", "duration"))

As we can see in above code I'm actually adding data to ArrayStop for specific IF condition only and I want this to be accessed outside this rdd loop but I was unable to do so So I created a variable "listings" to store data which is actually taking all the rows whereas I want only those entries which are added in ArrayStop. So what is the best way to bring any array outside nested If-else block. This is different from this issue Scala spark, listbuffer is empty Thanks,

Pinnacle
  • 165
  • 2
  • 14
  • why don't you create ArrayStop inside map function itself ? – Ramesh Maharjan May 09 '18 at 06:11
  • @RameshMaharjan - If I'll create ArrayStop inside map function will I be able to access its content outside ? and how exactly will help it me if I create ArrayStop inside ? – Pinnacle May 09 '18 at 07:29
  • inside map function but outside if else statement. try it and see it yourself. – Ramesh Maharjan May 09 '18 at 07:46
  • I tried but now it is considering separate array for each if block like I have three part in this "IF" as you can see I'm using add inside [else if (2nd part) (p._2 > 0 && p._2 != lastindex) {ArrayStop.add(imeistoppage(p._1.imei, lat, long, timeDiff))}] and last else block (3rd part). So now when I tried printing I got only array of last else block(3rd part) and not from above else if block i.e. 2nd part. Where I'm going wrong ? – Pinnacle May 09 '18 at 07:52
  • Possible duplicate of [Scala spark, listbuffer is empty](https://stackoverflow.com/questions/40699432/scala-spark-listbuffer-is-empty) – Alper t. Turker May 09 '18 at 10:24
  • 2
    Take a look on `flatMap()` instead of `map()` [here](https://stackoverflow.com/questions/22350722/what-is-the-difference-between-map-and-flatmap-and-a-good-use-case-for-each) is the difference and [`accumulators`](https://spark.apache.org/docs/2.2.0/rdd-programming-guide.html#accumulators). Both these things can help you achieve what you want. – Vladislav Varslavans May 09 '18 at 10:26
  • @VladislavVarslavans - this gives me an error type mismatch; found : java.util.ArrayList[imeistoppage] required: TraversableOnce[?] – Pinnacle May 11 '18 at 07:41
  • @user9613318 - my issue is somewhat different, I can relate to it but its different like I'm stuck inside nested If Else block and your mentioned link gives simple single block .. are you getting ? – Pinnacle May 11 '18 at 07:42
  • @Pinnacle. This is because `ArrayList` is java type, but in `flatMap()` is required scala type `TraversableOnce[T]`. Please refer [this](https://stackoverflow.com/questions/16162090/how-to-convert-a-java-util-list-to-a-scala-list) question to see how to convert java type to scala type. You can either convert your ArrayList to scala List, or just use scala List instead of ArrayList from start. – Vladislav Varslavans May 11 '18 at 09:49
  • Ok this resolved my issue. Can you please provide this comment as solution so that I can accept your answer and close the issue. Thanks, – Pinnacle May 11 '18 at 10:03

0 Answers0