How to access only spark logs in a yarn cluster

Question

I'm using spark with yarn in cluster mode and I wouldlike to retrieve only the logs concerning spark in order to use the history server. Is there a way to do that instead of copy pasting it from the big log files ?

`yarn application -list | grep "SPARK" | awk '{print $1}'` to get the applicationId of all spark applications — philantrovert, Sep 07 '17 at 08:45
@Tiffany Hatsune Hi, can you please help me setting zeppelin on spark cluster mode using docker. I am stuck with this for a while. Followed few tutorial on internet but still unsuccessful. — Rohit Barnwal, Sep 11 '17 at 07:18
Already asked a question here, https://stackoverflow.com/questions/46149472/running-zeppelin-on-spark-cluster-mode — Rohit Barnwal, Sep 11 '17 at 07:18

score 3 · Accepted Answer · edited Sep 08 '17 at 14:13

3

Yes, you can take logs of each task. You need to go to the yarn UI (8088 port) and copy id of the task. Then use command:

yarn logs --applicationId <TASK_ID>

For example:

yarn logs --applicationId application_1504771895665_0001

By this command you can see logs of each container that task used.

edited Sep 08 '17 at 14:13

Andrew Tobilko

48,120
14
91
142

answered Sep 07 '17 at 08:10

Yehor Krivokon

837
5
17

Is this command supposed to fetch the logs even after the job has finished or even failed/killed? – matanster Sep 05 '19 at 05:52

How to access only spark logs in a yarn cluster

1 Answers1