4

I'm new to NiFi. I'm trying to execute a Python script using ExecuteScript processor. When I tried a simple script which has no import commands it ran fine and showed output in nifi.StdOut. When I tried to run a script which includes import commands like import pandas. It showing the below error:

Import Error: No module named Pandas

I tried providing the path of the pkgs in the Module directory in properties. But it doesn't workout. Any help would be appreciated!

isherwood
  • 58,414
  • 16
  • 114
  • 157
Vicky
  • 312
  • 2
  • 9
  • 19

1 Answers1

4

I believe the issue is that pandas is a natively-compiled module (it is written in C) rather than being pure Python. The reason this is a problem is that due to the JSR-223 engine, the Apache NiFi ExecuteScript processor uses Jython rather than actual Python. So Python code is fine to run, but it can't depend on modules that aren't pure Python.

The workaround is to use the ExecuteStreamCommand processor to invoke the Python script which depends on pandas via the command-line (i.e. python my_script_that_uses_pandas.py). The flowfile content will be streamed to STDIN and captured from STDOUT. Here's a related answer describing this in detail.

Andy
  • 13,916
  • 1
  • 36
  • 78
  • Hi @Andy. I am stuck since many days in this problem: https://stackoverflow.com/questions/55606681/how-to-read-files-from-getfilesprocessor-in-nifi Could you please help me with that – whatsinthename Apr 14 '19 at 17:10
  • It looks like you are continuing to put new problems into the same question, which isn't how the site works. I suggest you open a new question with the explicit problem you are encountering. I think the problem is that the flowfile content is passed via `STDIN`, not put into a specific file location which you are trying to reference in your Python code. – Andy Apr 15 '19 at 02:36
  • Here you go: https://stackoverflow.com/questions/55682297/python-script-not-working-in-nifi-using-executestreamcommand-processor . – whatsinthename Apr 15 '19 at 03:39
  • Hey Andy, can you help me with the above post. Need some guidance over there. – whatsinthename Apr 17 '19 at 09:53