How to run Scala class in python?

Question

There is Scala object

 object Test {
  def t1(): String = {
    "test"
  }
}

need execute its in Python

Test.t1()

need execute not in JVM interpretator or convert Scala code to Python

Answer 1

Compile your scala into a jar and treat it as a java api. Typical options are py4j: https://www.py4j.org/, Jython: http://www.jython.org/, and JPipe.

If it is not a super frequent call (say, less than 1 million times per sec), I would go for py4j. If your python code only does basic things, you can use Jython as it is intrinsically Java world in which you will lose the whole lot eco-system of python. I wouldn't recommend JPipe too much as it ties python and jvm too closely: if one causes issues to the other, the entire program fails.

How does py4j work?

py4j requires a py4j server (a piece of software) that runs to talk to the APIs in your jar. Typically the server runs on the same machine as your python script. Then you can submit your query (the API call) to the py4j server and the server responds the results of the API calls. Although behind the scene it is "submitting a query to another server," py4j does a good job wrapping things, so it feels like a local python API.

This is also the method that Spark uses to enable pyspark integration.

Of course, it comes with an overhead, as you can imagine it goes to the network level (though local) with additional payload sizes in memory. You can see the discussion here: Py4J has bigger overhead than Jython and JPype