6

I am calling an API in my UDF and trying to log the output to a Logger and am getting serialization error.

Below is my Logger initialization code:

log4jLogger = spark._jvm.org.apache.log4j
LOGGER = log4jLogger.LogManager.getLogger(__name__)

In my UDF, I am logging using

LOGGER.info("Message")

But I am getting the error

pickle.PicklingError: Could not serialize object: Py4JError: An error occurred while calling o31.__getnewargs__. Trace:
py4j.Py4JException: Method __getnewargs__([]) does not exist

While registering the UDF

distance_udf = udf(distfunc, DoubleType())

Can you please correct me on what I should modify in my logging and also what if I want to log into a separate log file.

Thanks

Alper t. Turker
  • 34,230
  • 9
  • 83
  • 115
Pramod Sripada
  • 241
  • 1
  • 5
  • 16
  • 1
    This won't help you - [Calling Java/Scala function from a task](https://stackoverflow.com/q/31684842/8371915) - but explains why your code cannot work. – Alper t. Turker Jan 25 '18 at 20:40

0 Answers0