0

I am following this scala for data science book and git cloned all the sample codes from its repo. And when I first do sbt console and type in import breeze.lianlg._ command, it works fine. But if I ran a script that starts with a line import breeze.linalg._, the script could not run thru and the error messages indicate the importing is not successful. How do I fix this and make the script run? Thanks!

Edit: this is how the build.sbt file looks:

name := "S4DS"

organization := "s4ds"

version := "0.1.0-SNAPSHOT"

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.scalanlp" %% "breeze" % "0.11.2",
  "org.scalanlp" %% "breeze-natives" % "0.11.2",
  "org.slf4j" % "slf4j-simple" % "1.7.5"
)

This is a screen shot after I tried the CLASSPATH method: enter image description here

Jay Traband
  • 17,053
  • 1
  • 23
  • 44
user3768495
  • 4,077
  • 7
  • 32
  • 58

2 Answers2

2

Problem

The problem is that when you run console from SBT you're inside the project space, which includes your libraries.
But when you execute the script on its own, it is in the system space, and the libraries aren't present.

How to fix

Couple of options here:

  • Create an Uber JAR, instead of an script.
  • Add those libraries to your CLASSPATH.
  • Use Ammonite.

Uber JAR

The most common solution to this problem in general is to create an Uber JAR. It is also the most portable one, because now you have a jar with all its dependencies included, so you can run it anywhere - Also, because it is a Jar, you don't even need Scala anymore, just a JRE.
If you're going this way, check sbt-assembly. However, since you have probably a small script, going this way could be an overkill to your problem - but keep in mind that for larger projects this is the canonical solution.

CLASSPATH

The CLASSPATH env variable indicates Java where to search for classes (usually contained in libraries - Jars), so you only need to update this variable to include the missing libraries, you could do this by creating a folder (e.g. /opt/jars/) and downloading all jars to that folder, and then adding a line like one this in your .bashrc file, to make sure the env var is updated every time yo open a shell.

export CLASSPATH="/opt/jars/*:$CLASSPATH"

PS: Since SBT already downloaded the jars to the .ivy2/cache/ folder, you could point your CLASSPATH to that directory, but take care with multiple versions of the same Jar.

Ammonite

Ammonite is like a Scala RELP with super powers, one of its main advantages is to import Jars from ivy, which is exactly what you need. Also Ammonite can be used for scala scripts. So probably worth it giving a try.

Luis Miguel Mejía Suárez
  • 21,686
  • 3
  • 28
  • 45
  • Thank you very much! I am going with the CLASSPATH method since it seems to be the easiest to me. However I still can't make it work. Maybe it is because the library is not in the `cache` folder. I can't really tell. Please kindly see the screenshot. – user3768495 Sep 25 '18 at 00:03
  • Hi @user3768495, it is inside the `cache` folder, you can see there is a sub-folder `org.scalanlp`, inside that folder there should be subfolder `breeze` & `breeze-natives`, and inside those should be the jars. – Luis Miguel Mejía Suárez Sep 25 '18 at 00:11
  • Hi @Luis, yes they are in the `org.scalanlp` subfolder. But the problem still exist after I did `export CLASSPATH="~/.ivy2/cache/org.scalanlp/"`. Thoughts? – user3768495 Sep 25 '18 at 00:28
  • btw I am running it on MacOS. Does that matter? – user3768495 Sep 25 '18 at 00:29
  • Hi @user3768495, oh, sorry I wasn't clear enough, you should search inside those sub-sub-sub-folders until you find the Jar. e.g. `.ivy2/cache/org.antlr/antlr4/jars/antlr4-4.7.1.jar`. And that's the path you should add to your classpath. – Luis Miguel Mejía Suárez Sep 25 '18 at 00:40
  • I saw 3 subfolders under the `org.scalanlp` folder, which are `breeze-natives_2.11`, `breeze-macros_2.11`, and 'breeze_2.11`. All of them has a `jars/` folder, where I can find an actual `.jar` file. However I had no luck after trying all of them. Still got error `not found: object breeze`. – user3768495 Sep 25 '18 at 04:10
  • Uhm... I don't know what else could be, maybe try using the `-cp` [scala option](https://alvinalexander.com/scala/set-classpath-scala-shell-script) instead of the env var _(but they are supposed to be equivalent)_. Or maybe **breeze** has its own dependencies, and you should add them too to the classpath, and so on. Or maybe they need some compilation magic using macros _(which I don't know if they work on a script)_. The classpath is very tricky and that's why people usually don't use. I really can't help you more. – Luis Miguel Mejía Suárez Sep 25 '18 at 12:26
0

When you are in sbt console the libraries are taken from the cache but when you are running the script it will only check default libraries set. Please search the jars in .ivy cache. Copy the path and set the same in classpath then execute the script.

maxmithun
  • 1,089
  • 9
  • 18