0

I have a PySpark project where to log microbatch metrics, we have a jar which is passed using --jars, it works but I need to add a on/off switch for logging. How can I pass this on off switch when running using spark submit.

user2552949
  • 1
  • 1
  • 3
  • Does this answer your question? [How to turn off INFO logging in Spark?](https://stackoverflow.com/questions/25193488/how-to-turn-off-info-logging-in-spark) – teedak8s Aug 30 '22 at 06:18
  • Won't work! as I am using cloudwatch to log some data - Ex. metricsLogger.putMetric("addBatch", progressEvent.durationMs().get("addBatch"), Unit.MICROSECONDS); – user2552949 Aug 30 '22 at 11:14

0 Answers0