2

I have a native library (Freeling), which I have compiled using cmake and make natively and installed through cluster startup actions (so, it should exist in master and each worker)

Even so, I'm getting this error when calling System.loadLibrary

Exception in thread "main" java.lang.UnsatisfiedLinkError: no Jfreeling in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)

I've tried using the following properties to get the program to find the library (It's called in a static block of code)

      "properties": {
        "spark.driver.extraClassPath": "/usr/local/share/freeling/APIs/java/Jfreeling.jar:/usr/local/lib/libfreeling.so",
        "spark.executor.extraClassPath": "/usr/local/share/freeling/APIs/java/Jfreeling.jar:/usr/local/lib/libfreeling.so",
        "spark.executor.extraLibraryPath": "/usr/local/lib/libfreeling.so",
        "spark.driver.extraLibraryPath": "/usr/local/lib/libfreeling.so",
        "spark.executorEnv.LD_PRELOAD": "/usr/local/lib/libfreeling.so",
        "spark.yarn.dist.files": "/usr/local/lib/libfreeling.so",
        "spark.yarn.appMasterEnv.LD_PRELOAD": "libfreeling.so",
        "spark.files": "/usr/local/lib/libfreeling.so",
        "spark.executorEnv.LD_LIBRARY_PATH": "libfreeling.so"
      },
      "jarFileUris": [
        "file:///usr/local/share/freeling/APIs/java/Jfreeling.jar",
        "file:///usr/local/lib/libfreeling.so"
      ],
efaj
  • 925
  • 1
  • 12
  • 23

2 Answers2

1

Can you try putting your library under /usr/lib/hadoop/lib/native/? In /etc/spark/conf/spark-env.sh, it has

# Spark got rid of SPARK_LIBRARY_PATH in 1.0
# It has properties for extraLibraryPaths, but this is more extensible
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
Dagang
  • 24,586
  • 26
  • 88
  • 133
0

You should add /usr/local/share/freeling/APIs/java/Jfreeling.jar to your CLASSPATH, and /usr/local/share/freeling/APIs/java/libJfreeling.so to your LD_LIBRARY_PATH.

Lluís Padró
  • 215
  • 1
  • 5