0

Learning Scala from the Scala for Data Science book and the companion Github repo, here I am particularly talking about the build file for Chapter 2, copied below (with minor modification) for reference.

name := "Logistic_regression"

organization := "My Organisation"

version := "0.1.0-SNAPSHOT"

scalaVersion := "2.11.7" // 3.2.2 does not work

libraryDependencies ++= Seq(
  "org.scalanlp" %% "breeze" % "0.11.2",
  "org.scalanlp" %% "breeze-natives" % "0.11.2",
  "org.slf4j" % "slf4j-simple" % "1.7.5"
)

Being new to scala from python and C++, the idea of how different scala environments, dependencies etc. work is not entirely clear to me yet, but can I think of the build.sbt as a form of Dockerfile, that can take care of the dependencies on any host by creating its own sandbox?

The repo itself uses scalaVersion := "2.11.7" which works fine. But I wanted to match it to my system scala version (see below for the version details).

della@dell-xps ~/s/l/s/chap02 (master)> scala --version
Scala code runner version 3.2.2 -- Copyright 2002-2023, LAMP/EPFL
della@dell-xps ~/s/l/s/chap02 (master)> sbt --version
sbt version in this project: 1.8.2
sbt script version: 1.8.2

But changing it to 3.2.2 results in the following stack-trace (just the top part).

[warn]  Note: Unresolved dependencies path:
[error] sbt.librarymanagement.ResolveException: Error downloading org.scalanlp:breeze_3:0.11.2
[error]   Not found
[error]   Not found
[error]   not found: /home/della/.ivy2/localorg.scalanlp/breeze_3/0.11.2/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/scalanlp/breeze_3/0.11.2/breeze_3-0.11.2.pom

So why is it that build.sbt cannot resolve the dependencies for a more recent scala version? Or, should it?

Further, if I want to work on the same project with the newer scala version, should I do it without sbt, and somehow installing the dependencies at the system level, available to all projects? How do I go around it?

Dmytro Mitin
  • 48,194
  • 3
  • 28
  • 66
Della
  • 1,264
  • 2
  • 15
  • 32

1 Answers1

3

can I think of the build.sbt as a form of Dockerfile, that can take care of the dependencies on any host by creating its own sandbox?

There is no sandboxing here in Docker sense, you're not creating an isolated environment (@SethTisue commented that this can be considered as sandboxing in the sense that sbt manages dependencies itself, rather than relies on the ones installed systemwide). This is using a build tool like Sbt, Mill, Fury, Cbt in Scala, Maven or Gradle in Java, Bazel in Java/C++/Go, Pants in Python/Go/Java/Scala/Kotlin, Make in C++, Pip in Python, Cabal or Stack in Haskell etc.

Scala 2 and Scala 3 are quite different

https://docs.scala-lang.org/scala3/reference/

http://dotty.epfl.ch/

Scala 3 is still quite new version of Scala. Not all dependencies are published for Scala 3. Not all of them can be easily migrated into Scala 3 (because of Scala 2 macros etc.).

Scala 2.10, 2.11, 2.12, 2.13 are binary incompatible. That's why we use %% for Scala libraries: "org.scalanlp" %% "breeze" % "0.11.2" is "org.scalanlp" % "breeze_2.11" % "0.11.2" while using scalaVersion := "2.11.7". But there is Scala 2.13 - Scala 3 interop

https://docs.scala-lang.org/scala3/guides/migration/compatibility-intro.html

There is compatibility mode CrossVersion.for3Use2_13/CrossVersion.for2_13Use3

https://docs.scala-lang.org/scala3/guides/migration/compatibility-classpath.html

How to use akka in Scala 3?

You're using quite old versions of your dependencies. As you can see, Breeze and Breeze-natives 0.11.2 were published for Scala 2.10, 2.11. They are published for Scala 3 since Breeze 2.0

https://mvnrepository.com/artifact/org.scalanlp/breeze

https://mvnrepository.com/artifact/org.scalanlp/breeze-natives

So try the following build.sbt

scalaVersion := "3.2.2"

libraryDependencies ++= Seq(
  "org.scalanlp" %% "breeze"         % "2.1.0",
  "org.scalanlp" %% "breeze-natives" % "2.1.0",
  "org.slf4j"    %  "slf4j-simple"   % "2.0.7"
)

For Breeze 1.x there is no Scala-3 version but there is Scala-2.13 version, so you can try compatibility mode

scalaVersion := "3.2.2"

libraryDependencies ++= Seq(
  "org.scalanlp" %% "breeze"         % "1.0" cross CrossVersion.for3Use2_13,
  "org.scalanlp" %% "breeze-natives" % "1.0" cross CrossVersion.for3Use2_13,
  "org.slf4j"    %  "slf4j-simple"   % "2.0.7"
)

But before Breeze 1.0 there was no Scala-2.13 version, so you can use only Scala 2.10-2.12 with them

scalaVersion := "2.12.17"

libraryDependencies ++= Seq(
  "org.scalanlp" %% "breeze"         % "0.13.2",
  "org.scalanlp" %% "breeze-natives" % "0.13.2",
  "org.slf4j"    %  "slf4j-simple"   % "2.0.7"
)

If you're using sbt, most probably you will not need to install anything in your system (except sbt and Java). Sbt will resolve dependencies itself.

If you installed something in the system, most probably this is irrelevant for sbt builds.

Dmytro Mitin
  • 48,194
  • 3
  • 28
  • 66
  • Thanks a lot for the detailed answer. Your first suggestion worked. But if you do not mind some follow up 1. Where does sbt get versions like 2.11 or 2.12 when ```scala --version``` is 3.2.2 from my terminal (Ubuntu 22.04)? Does SBT mange all scala versions inside itself? Similarly, if one project needs breeze 2.1.0 and another one needs breeze 1.9.3 etc.? 2. Does scala violate semantic versioning where a minor version upgrade should not break the API, as it is followed in libraries like tensorflow or numpy? I am just a bit surprised if this is the case. – Della Mar 30 '23 at 05:02
  • 3. Your first suggestion of the build.sbt works but needs modifying some *.scala codes as it seems breeze changed their vector inner product APIs. But those modifications trigger warning from my IDE (intellij). So intellij uses some other version of breeze to inspect my code, which is like the system-wide version? But I do not remember installing any breeze at the system level, all I did was modifying the build.sbt files. – Della Mar 30 '23 at 05:03
  • 4
    1/ SBT doesn't use the Scala installed on your system. It doesn't event need Scala installed. It will download necessary Scala version when needed. – Gaël J Mar 30 '23 at 06:26
  • 3
    2/ Scala versioning is not semver indeed. 2.x is more 2.major.minor. 3.x is closer to semver. You can learn more about Scala 3 versioning scheme on the official website/blog. – Gaël J Mar 30 '23 at 06:29
  • 2
    3/ you probably need to tell IntelliJ to reload the project (SBT tab, there's a reload button for instance) – Gaël J Mar 30 '23 at 06:30
  • 2
    While it isn't sandboxing in the stronger Docker sense, but you could call it “sandboxing” in the sense that an sbt build specifies all of its own dependencies and their versions, rather than relying on anything being installed systemwide (except a JVM and the sbt launch script). – Seth Tisue Mar 30 '23 at 08:29
  • 1
    @Della *"Where does sbt get versions like 2.11 or 2.12 when `scala --version` is 3.2.2 from my terminal (Ubuntu 22.04)?"* Once again, `scala --version` from your system is irrelevant to sbt. Relevant is `scalaVersion := "3.2.2"` in `build.sbt`. *"Does SBT mange all scala versions inside itself?"* Yes, it will download all necessary versions of Scala standard library, Scala compiler, sbt plugins, compiler plugins, dependencies etc. Different subprojects of your project can use different versions of Scala. – Dmytro Mitin Mar 31 '23 at 04:57
  • 1
    @Della Sbt needs different versions of Scala to compile `build.sbt` itself and your Scala sources. You can cross-build your project with several versions of Scala. In my system I can see downloaded jars at `~/.cache/coursier`, `~/.ivy2/cache`, `~/.sbt`. Moreover, sbt even manages different versions of sbt itself because your system sbt and sbt for your project (specified in `project/build.properties`) can be different. – Dmytro Mitin Mar 31 '23 at 04:57
  • @Della *"if one project needs breeze 2.1.0 and another one needs breeze 1.9.3 etc.?"* I guess there is no Breeze 1.9.3 https://mvnrepository.com/artifact/org.scalanlp/breeze Out of Breeze 2.0 and 2.1.0 sbt selects 2.1.0, out of 1.3 and 2.1.0 it selects 2.1.0. https://stackoverflow.com/questions/50480457/how-sbt-resolves-conflicted-version-of-libraries-in-dependency https://www.scala-sbt.org/1.x/docs/Library-Management.html#Conflict+Management – Dmytro Mitin Mar 31 '23 at 05:36
  • 1
    @Della Regarding Scala versioning and semver you can read discussions at https://news.ycombinator.com/item?id=6678471 https://contributors.scala-lang.org/t/scala-versions-are-inconsistent-with-semver/5264 – Dmytro Mitin Mar 31 '23 at 05:39
  • @Della *"I do not remember installing any breeze at the system level"* Once again, you don't need to install it in the system. Or if you did this is irrelevant to sbt builds. IntelliJ can either manage dependencies and build project itself (IntelliJ Scala project) or use system sbt to manage dependencies and build project (Intellij sbt project). It's better to configure IntelliJ to use sbt. – Dmytro Mitin Mar 31 '23 at 05:49
  • @Della When IntelliJ shows warnings, compile errors highlighted in red etc. it uses its own internal compiler ([presentational compiler](https://stackoverflow.com/questions/9468367)) different from the standard one. So you should get used to eventually rely on `sbt clean compile`. When you modify `build.sbt` don't forget to re-import sbt project in IntelliJ. – Dmytro Mitin Mar 31 '23 at 05:49
  • https://stackoverflow.com/questions/75531849/why-does-sbt-show-a-different-scalaversion-than-my-system – Dmytro Mitin Apr 04 '23 at 06:06
  • https://docs.scala-lang.org/scala3/reference/language-versions/source-compatibility.html https://docs.scala-lang.org/scala3/reference/language-versions/binary-compatibility.html – Dmytro Mitin Apr 05 '23 at 02:12
  • https://degoes.net/articles/new-scala-build-tool – Dmytro Mitin Apr 12 '23 at 13:03