0

When I use this code in a notebook or a job:

import logging


def initialize_logging(log_level) -> logging:
    logging.basicConfig(format='%(asctime)s,%(msecs)03d %(levelname)-8s %(name)s %(message)s',
                        datefmt='%Y-%m-%d %H:%M:%S', level=log_level)


initialize_logging(logging.INFO)
logging.info("some message")

I get this log:

2021-09-01 17:34:12,731 INFO     root some message
2021-09-01 17:34:12,747 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,757 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,757 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,759 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,759 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,760 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,760 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,796 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,799 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,799 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,799 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,800 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,801 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,802 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,802 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,803 INFO     py4j.java_gateway Received command c on object id p0
2021-09-01 17:34:12,803 INFO     py4j.java_gateway Received command c on object id p0

I want to get only my message because my log is full with this 2021-09-01 17:34:12,803 INFO py4j.java_gateway Received command c on object id p0

Borislav Blagoev
  • 187
  • 5
  • 15

1 Answers1

3

Try the below (set the log level of py4j to error)

logger = logging.getLogger("py4j")
logger.setLevel(logging.ERROR)
balderman
  • 22,927
  • 7
  • 34
  • 52
  • This actually used to work for me, but now the solution is invalid. Also, this may be a duplicate of this question (https://stackoverflow.com/questions/37252527/how-to-hide-py4j-java-gatewayreceived-command-c-on-object-id-p0), which includes similar answers, none of which are working for me now. Maybe there's been a change to Spark/PySpark causing new issues? – James Adams Nov 17 '22 at 20:10
  • 1
    I just checked it @JamesAdams and it's working for me. I am using Databricks on AWS. – Amey Joshi May 19 '23 at 08:27