4

I'm searching for a way, to forward my driver's MDC to master and workers, such that I can identify messages stemming from the same operation? E.g. I have a application dependent query/job-id which I want to have attached to structured log messages in my spark workers.

  • Impossible?
  • Undocumented?
  • Easy, but I was to dumb to find?

Regards.

Anuj Mehra
  • 320
  • 3
  • 19
Joachim Rosskopf
  • 1,259
  • 2
  • 13
  • 24

1 Answers1

3

If I understand the PR correctly (I am new to Spark) then since version 3.1 it should be now possible to add custom entries to MDC via spark.sparkContext.setLocalProperty("mdc." + name, "value").

vit fiala
  • 68
  • 6
  • 1
    this does not work for all spark logs, for isntance spark sql loggers and org.apache.spark.scheduler.TaskSetManager or org.apache.spark.storage.ShuffleBlockFetcherIterator, those are examples where they don't seem to take the MDC set by the spark context – bachr Jul 14 '21 at 20:13
  • I have the same issue with TaskSetManager. Any idea how to get the MDC there? – Emile Sonneveld Apr 17 '23 at 16:41