Python in Spark Scala via rdd.pipe redirect log

Asked Jun 30 '18 at 07:53

Active Jul 01 '18 at 14:05

Viewed 287 times

I am not using PySpark . I am using scala to run my spark application. I execute a python script within the spark application by calling calling RDD.pipe. I want to be able to log something from the python script ,and see the logged information in the executor log. I am not using PySpark. I am executing a Python script within a Spark application via the API RDD.pipe.

I want to somehow provide the user the ability to debug the script by printing some messages at the level of the Python script and have those Python scripts available at the level of the Spark logger.

How can I do that ?

edited Jun 30 '18 at 17:19

asked Jun 30 '18 at 07:53

JavaPlanet

Possible duplicate of [PySpark logging from the executor](https://stackoverflow.com/questions/40806225/pyspark-logging-from-the-executor) – Alper t. Turker Jun 30 '18 at 10:28
I do not believe this is a duplicate @user8371915 – JavaPlanet Jul 01 '18 at 01:13

Python in Spark Scala via rdd.pipe redirect log

0 Answers0