1

Being aware that Range is slow compared to a loop under Scala2, I migrated the benchmark to Scala3, and observed a horrible downgrade in performance in case of Range class.

Note that while loop time is comparable .

My questions:

  • Is my benchmark correct?
  • Did Range implementation change under Scala3 and performance hit is to be expected?
Scala2:
while loop time: 0.015882699999999996 ms
range loop time: 0.20659249999999996 ms

Scala3:
while loop time: 0.011921900000000001 ms
range loop time: 464.97765689999994 ms

Benchmark code:

package scalashop

import org.scalameter.{Key, Warmer, config}


object TestDoubleLoopPerformance {
  val standardConfig = config(
    Key.exec.minWarmupRuns := 5,
    Key.exec.maxWarmupRuns := 10,
    Key.exec.benchRuns := 10,
    // Key.verbose := true
  ) withWarmer(Warmer.Default())


  def main(args: Array[String]): Unit = {
    val whileLoopTime = standardConfig measure {
      for (iter <- 0 to 100) {
        sumWithWhileLoop(1000, 2000)
      }
    }
    println(s"while loop time: $whileLoopTime")

    val rangeLoopTime = standardConfig measure {
      for (iter <- 0 to 100) {
        sumWithRange(1000, 2000)
      }
    }
    println(s"range loop time: $rangeLoopTime")
  }

  private def sumWithRange(xEnd: Int, yEnd: Int) = {
    var sum = 0
    for {
      x <- 0 until xEnd
      y <- 0 until yEnd
    } sum += 1
    sum
  }

  private def sumWithWhileLoop(xEnd: Int, yEnd: Int) = {
    var sum = 0
    var x = 0
    while (x < xEnd) {
      var y = 0
      while (y < yEnd) {
        sum += 1
        y += 1
      }
      x += 1
    }
    sum
  }
}

Environment:

  • JVM: openjdk 11.0.9.1
  • Scalameter: 0.21
Lesiak
  • 22,088
  • 2
  • 41
  • 65
  • Which version of Scala 2? Scala 3 and Scala 2.13 should use the same collection library, but it's somewhat different from Scala 2.12's. – Levi Ramsey Aug 26 '21 at 15:36
  • @LeviRamsey 2.13.0 vs 3.0.0 – Lesiak Aug 26 '21 at 15:39
  • 1
    I would take a look at the bytecode of the range benchmark (`:javap` in `sbt console`) to get an idea of if there's an optimization that dotty isn't applying. It's also not impossible that dotty emits marginally more bytecode and that causes the JIT to decide not to apply some optimization. – Levi Ramsey Aug 26 '21 at 16:08
  • 1
    (this also probably warrants an issue being raised against dotty, if for no other reason than to get guidance on collecting more details) – Levi Ramsey Aug 26 '21 at 16:09
  • 1
    @LeviRamsey I analysed javap output and submitted a bug to dotty: https://github.com/lampepfl/dotty/issues/13402 – Lesiak Aug 27 '21 at 08:31
  • Could you add your build.sbt file to the post? I can't get scalameter to work with scala3. sbt reports that it can't find the packages. – Michael W. Sep 20 '22 at 13:47
  • 1
    @MichaelW. You need to use CrossVersion setting `("com.storm-enroute" %% "scalameter-core" % "0.21").cross(CrossVersion.for3Use2_13)` See https://www.scala-lang.org/blog/2021/04/08/scala-3-in-sbt.html – Lesiak Sep 20 '22 at 15:05

0 Answers0