Where are logs in Spark on YARN?

Question

I'm new to spark. Now I can run spark 0.9.1 on yarn (2.0.0-cdh4.2.1). But there is no log after execution.

The following command is used to run a spark example. But logs are not found in the history server as in a normal MapReduce job.

SPARK_JAR=./assembly/target/scala-2.10/spark-assembly-0.9.1-hadoop2.0.0-cdh4.2.1.jar \
./bin/spark-class org.apache.spark.deploy.yarn.Client --jar ./spark-example-1.0.0.jar \
--class SimpleApp --args yarn-standalone  --num-workers 3 --master-memory 1g \
--worker-memory 1g --worker-cores 1

where can I find the logs/stderr/stdout?

Is there someplace to set the configuration? I did find an output from console saying:

14/04/14 18:51:52 INFO Client: Command for the ApplicationMaster: $JAVA_HOME/bin/java -server -Xmx640m -Djava.io.tmpdir=$PWD/tmp org.apache.spark.deploy.yarn.ApplicationMaster --class SimpleApp --jar ./spark-example-1.0.0.jar --args 'yarn-standalone' --worker-memory 1024 --worker-cores 1 --num-workers 3 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr

In this line, notice 1> $LOG_DIR/stdout 2> $LOG_DIR/stderr

Where can LOG_DIR be set?

score 45 · Answer 1 · edited Feb 14 '22 at 11:59

45

You can access logs through the command

yarn logs -applicationId <application ID> [OPTIONS]

general options are:

appOwner <Application Owner> - AppOwner (assumed to be current user if not specified)
containerId <Container ID> - ContainerId (must be specified if node address is specified)
nodeAddress <Node Address> - NodeAddress in the format nodename:port (must be specified if container id is specified)

Examples:

yarn logs -applicationId application_1414530900704_0003                                      
yarn logs -applicationId application_1414530900704_0003 myuserid

// the user ids are different
yarn logs -applicationId <appid> -appOwner <userid>

edited Feb 14 '22 at 11:59

Liran Funaro

2,750
2
22
33

answered Sep 28 '14 at 07:36

MARK

2,302
4
25
44

12

That's only true if `yarn.log-aggregation-enable` is `true` in `yarn-site.xml` _and_ the application is already finished. – thSoft Oct 27 '15 at 14:35
nish1013 Try yarn application --list – iec2011007 Jan 20 '16 at 08:07
how to get the logs if the log files are zipped ? I tried this command but it prints garbled zipped file output. thanks. – Ravi Chinoy May 11 '16 at 16:49
To get the application ID, run `yarn application -list -appStates ALL` and get the first field of the first line that starts with "application_". In my case it's something like "application_1480604706480_0001". – Ben Hoyt Dec 01 '16 at 17:28
@jacek Laskowski, From the spark history server, I am unable to access the container logs if log aggregation is enabled. it keeps checking in the node manager log directory and not in the aggregated log location. This is for a completed job – Harikrishnan Ck Jun 08 '17 at 02:07
does it matter if the spark job was client mode or cluster mode? I do not seem to see the whole logs (i am under client mode) – soMuchToLearnAndShare Nov 13 '17 at 11:52
How do you determine which container was running the driver? – JMess Jun 07 '19 at 20:19
what do I do if I want a running log of this application? – Aashish Chaubey Apr 05 '22 at 07:56

score 27 · Accepted Answer · edited Jul 04 '17 at 11:27

Pretty article for this question:

Running Spark on YARN - see the section "Debugging your Application". Decent explanation with all required examples.

The only thing you need to follow to get correctly working history server for Spark is to close your Spark context in your application. Otherwise, application history server does not see you as COMPLETE and does not show anything (despite history UI is accessible but not so visible).

score 23 · Answer 3 · answered Oct 07 '16 at 07:32

23

None of the answers make it crystal clear where to look for logs ( although they do in pieces) so I am putting it together.

If log aggregation is turned on (with the yarn.log-aggregation-enable yarn-site.xml) then do this

yarn logs -applicationId <app ID>

However, if this is not turned on then one needs to go on the Data-Node machine and look at

$HADOOP_HOME/logs/userlogs/application_1474886780074_XXXX/

application_1474886780074_XXXX is the application id

answered Oct 07 '16 at 07:32

Somum

2,382
26
15

2

For those like me who do not know how to get the application id: use `yarn applications -list` – stefan.m Mar 20 '17 at 09:06
2

@stefan.m, that would be `yarn application -list` and not `yarn applications -list` – iruvar Aug 22 '17 at 12:20
2

And `yarn logs -applicationId -log_files stdout` will retrieve just the stdout if that's all your interested in:). – snark Nov 27 '17 at 16:45
For future readers, I found this library useful. Please give it a try : https://github.com/hammerlab/yarn-logs-helpers – user238607 Mar 11 '21 at 20:37

score 15 · Answer 4 · edited Mar 02 '17 at 18:39

15

It logs to:

/var/log/hadoop-yarn/containers/[application id]/[container id]/stdout

The logs are on every node that your Spark job runs on.

edited Mar 02 '17 at 18:39

rado

4,040
3
32
26

answered Jul 15 '15 at 23:10

Carlos Rendon

6,174
5
34
50

1

When log aggregation isn't turned on it's stored in `/tmp/logs`. I found mine using `hdfs dfs -ls /tmp/logs/{USER}/logs` where `{USER}` is the user which launched the spark application. – alex Jul 11 '19 at 18:33
This worked for me. ALSO, if you are running a cluster, the logs might be on one of the nodes. So make sure you cheap each node in your cluster and see if the logs are in this directory structure. – mawaldne Sep 25 '20 at 19:52

Where are logs in Spark on YARN?

4 Answers4

Linked