systemML:'JavaPackage' is not callable. when I use MLContext

Question

I follow the tutorial from the website,I choose the "Sample notebook"--Deep Learning Image Classification,(look here) and when I run ml = MLContext(sc), errors occur as follow:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/.local/lib/python3.5/site-packages/systemml/classloader.py in createJavaObject(sc, obj_type)
     86     try:
---> 87         return _createJavaObject(sc, obj_type)
     88     except (py4j.protocol.Py4JError, TypeError):

~/.local/lib/python3.5/site-packages/systemml/classloader.py in _createJavaObject(sc, obj_type)
     46     if obj_type == 'mlcontext':
---> 47         return sc._jvm.org.apache.sysml.api.mlcontext.MLContext(sc._jsc)
     48     elif obj_type == 'dummy':

TypeError: 'JavaPackage' object is not callable

During handling of the above exception, another exception occurred:

Py4JJavaError                             Traceback (most recent call last)
<ipython-input-5-724894051c3e> in <module>()
----> 1 ml = MLContext(sc)

~/.local/lib/python3.5/site-packages/systemml/mlcontext.py in __init__(self, sc)
    697             raise ValueError("Expected sc to be a SparkContext or SparkSession, got " % str(type(sc)))
    698         self._sc = sc
--> 699         self._ml = createJavaObject(sc, 'mlcontext')
    700 
    701     def __repr__(self):

~/.local/lib/python3.5/site-packages/systemml/classloader.py in createJavaObject(sc, obj_type)
     92         # First load SystemML
     93         jar_file_name = _getJarFileName(sc, '')
---> 94         x = _getLoaderInstance(sc, jar_file_name, 'org.apache.sysml.utils.SystemMLLoaderUtils', hint + 'SystemML.jar')
     95         x.loadSystemML(jar_file_name)
     96         try:

~/.local/lib/python3.5/site-packages/systemml/classloader.py in _getLoaderInstance(sc, jar_file_name, className, hint)
     69         jar_file_url_arr[0] = jar_file_url
     70         url_class_loader = sc._jvm.java.net.URLClassLoader(jar_file_url_arr, sc._jsc.getClass().getClassLoader())
---> 71         c1 = sc._jvm.java.lang.Class.forName(className, True, url_class_loader)
     72         return c1.newInstance()
     73     else:

/usr/local/lib/python3.5/dist-packages/py4j/java_gateway.py in __call__(self, *args)
   1158         answer = self.gateway_client.send_command(command)
   1159         return_value = get_return_value(
-> 1160             answer, self.gateway_client, self.target_id, self.name)
   1161 
   1162         for temp_arg in temp_args:

~/software/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

/usr/local/lib/python3.5/dist-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    318                 raise Py4JJavaError(
    319                     "An error occurred while calling {0}{1}{2}.\n".
--> 320                     format(target_id, ".", name), value)
    321             else:
    322                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:java.lang.Class.forName.
: java.lang.ClassNotFoundException: org.apache.sysml.utils.SystemMLLoaderUtils
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:280)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:748)

It seems that it cannot find SystemML.jar? I download the systemml binary files maybe include this file.But I don't know what I can do. Thanks! Thanks!!!

score 2 · Answer 1 · answered Dec 03 '17 at 05:24

2

As you correctly pointed out, the error occurs because the SystemML classes were not found. There are two ways to resolve this:

(Recommended) Instead of downloading python files, I would suggest installing SystemML via pip: either pip install systemml (released version) or pip install https://sparktc.ibmcloud.com/repo/latest/systemml-1.0.0-SNAPSHOT-python.tar.gz (latest code). This takes care of packaging the necessary jar.
If you prefer using downloaded python files, then you will have to provide SystemML.jar and systemml-*-extra.jar via --driver-class-path option of pyspark.

answered Dec 03 '17 at 05:24

Niketan

156
2
8

Thanks for your answer! But when I input `import systemml`, the error doesn't occur.I think it means I have installed SystemML correctly.And when I add run the code `import systemml as sml import numpy as np from pyspark.context import SparkContext from pyspark.sql.session import SparkSession sc = SparkContext('local') m1 = sml.matrix(np.ones((3,3)) + 2) print(m1.sum(axis=1).toNumPy())` the error changed as follows: `INFO SparkContext: Added JAR **/systemml-0.15.0-extra.jar at **/systemml-0.15.0-extra.jar *** TypeError: 'JavaPackage' object is not callable` – C.Y.Liao Dec 05 '17 at 07:21
Since extra jar is added first, you are likely hitting the bug that was fixed recently by the commit: https://github.com/apache/systemml/commit/696fe3f6d0b57228f98e0959dc4de8b52ea0d6ed The bug in this commit occurred due to the order in which the jars were loaded (which depended earlier on os.listdir). To use previous versions (ie before the above mentioned commit), you have to use option 2 described in my previous comment. – Niketan Dec 06 '17 at 15:46

systemML:'JavaPackage' is not callable. when I use MLContext

1 Answers1