0

If someone can explain the limitation of Jython in the context of UDF in Pig, that would be great. Here are key questions:

  • Can it run in both Hadoop 1.x and YARN?
  • Does it support Python 3.4? From python UDF version with Jython/Pig, I guess it is unlikely but the post is one year plus old.
  • Is there any limitation in terms of modules supported?

Essentially I am trying to see if it makes sense to use Jython (preferably in Python 3)

Thanks!

Community
  • 1
  • 1
kee
  • 10,969
  • 24
  • 107
  • 168

1 Answers1

1

Here is a link from the FAQ. I can't answer the question on Hadoop or yarn.

Jython does not support Python 3. The latest version supports python 2.7.0.

Many of python's scientific modules are written in C or Cython. For example, SciPy uses a fortran project BLAS for linear algebra. C code was written that forms the bridge between Python and BLAS. Java byte code is not compatible with native binary.

There is a project JyNI. Located here. It attempts to bridge the gap. I don't know how mature it is.

According to jython's web page, they are attempting to make jython compatible with the python extensions api. As of right now, there has not been official release, and I assume it is still in alpha.

Java does not have a way for integrating native code into Java programs as of Java 8. I assume they are doing hackish stuff to get it to work.

yxre
  • 3,576
  • 21
  • 20
  • Java *does* have an official way to integrate native code. It is called JNI (http://docs.oracle.com/javase/8/docs/technotes/guides/jni/). Based on this there are tools like JNA or SWIG that allow a somewhat more comfortable use. The Graal Java VM additionally features GNFI, a true alternative to JNI that is said to be much more efficient (but is however Graal-VM only). – stewori Jun 02 '15 at 01:51