0

I have build a scala application and loggin with log4j. Sample code: 

logger.info(“This gets printed in client mode  but not in cluster mode”)

Issue is when i run in yarn-client mode above statements gets printed, but when I run yarn-cluster mode it doesn’t.

I want such messages to be printed in cluster mode as well.

Already tried

Spark-submit —files option

and setting spark.executor.extrajavaoptions and spark.driver.extrajavaoptions`

franiis
  • 1,378
  • 1
  • 18
  • 33
Abhishek Mishra
  • 21
  • 8
  • where do you want to have your statements printed? AFAIk they are printed on the driver (unless you log inside transformations/actions, in this case they are printed on the executors). If you switsch from yarn-client to yarn-cluster, the driver machine will change (in yarn-client, the machine where you submit the application is the driver, in yarn-cluster this is not the case) – Raphael Roth Oct 13 '17 at 08:55
  • Hi Raphael, Thanks. I want these statement to be printed on console or in a log file in cluster mode. How can that be done? – Abhishek Mishra Oct 13 '17 at 09:16
  • @AbhishekMishra: It should be available in the log file, you can access the log files using `yarn logs -applicationId ` – Shankar Oct 13 '17 at 10:32
  • @AbhishekMishra : you can also look at the https://stackoverflow.com/questions/23058663/where-are-logs-in-spark-on-yarn question for more – Shankar Oct 13 '17 at 10:34
  • Thanks Shankar. Can we redirect these yarn logs to a different file? Actually i want to generate these log at a specific location. – Abhishek Mishra Oct 13 '17 at 10:58

0 Answers0