How to send out logs from a UDF function in Pyspark

Question

If you add any kind of logging into a UDF function in PySpark, it won't appear anywhere. Is it some kind of method to make this happen?

So far I tried standard python logging, py4j and also print.

We're running PySpark 2.3.2 with YARN cluster manager on AWS EMR clusters.

For example. Here's a function I want to use:

def parse_data(attr):
    try:
        # execute something
    except Exception as e:
        logger.error(e)
        return None

I convert it to UDF:

import pyspark.sql.functions as F
parse_data_udf = F.udf(parse_data, StringType())

And I will use it on a dataframe:

from pyspark.sql import types as pst
dataframe = dataframe.withColumn("new_column", parse_data_udf("column").cast(pst.StringType())

The logs from the function will NOT appear anywhere.

Possible duplicate of https://stackoverflow.com/questions/25407550/how-do-i-log-from-my-python-spark-script — Aasim Khan, Oct 15 '19 at 14:38
Look here: https://stackoverflow.com/questions/40806225/pyspark-logging-from-the-executor — Mariusz, Oct 15 '19 at 18:43
Both of these are about general logging, my question is about, logs inside of a UDF. — Géza Hodgyai, Oct 21 '19 at 11:47
Hi did you come up with any solution, im stuck at same point — Nrithya M, Apr 11 '21 at 20:58
i'm checking similar issue .. the output of the print statements in udf are in the stderr — Karan Alang, Oct 12 '22 at 20:26

score 0 · Answer 1 · answered Oct 22 '21 at 20:28

When using yarn you can use the following YARN CLI command to check the container logs.

This is where the stdout/stderr (and thus what you log inside the udf) is probably located.

yarn logs -applicationId <Application ID> -containerId <Container ID>

1 Answers1