how to capture the job status in shell script for spark-submit

Question

I am using bashshell with spark-sql-2.4.1v. I submit my spark job using spark-submit in a shell script.

Need to capture the status of my job. how can this be achieved ?

Anyhelp/advice please ?

Srinivas · Accepted Answer · 2020-06-05T05:52:29.697

3

Check below code.

process_start_datetime=$(date +%Y%m%d%H%M%S)
log_path="<log_dir>"
log_file="${log_path}/${app_name}_${process_start_datetime}.log"

spark-submit \
    --verbose \
    --deploy-mode cluster \
    --executor-cores "$executor_cores" \
    --num-executors "$num_executors" \
    --driver-memory "$driver_memory" \
    --executor-memory "$executor_memory"  \
    --master yarn \
    --class main.App "$appJar" 2>&1 | tee -a "$log_file"

status=$(grep "final status:" < "$log_file" | cut -d ":" -f2 | tail -1 | awk '$1=$1')

To get Application Id

applicationId=$(grep "tracking URL" < "$log_file" | head -n 1 | cut -d "/" -f5)

edited Jun 05 '20 at 05:52

answered Jun 05 '20 at 04:57

Srinivas

8,957
2
12
26

sir what is this 2>&1 | tee -a "$log_file" ?? – BdEngineer Jun 05 '20 at 05:19
so what is 2 here ? – BdEngineer Jun 05 '20 at 05:44
echo $LOG_PATH gives blank , so can i set this current dir ? – BdEngineer Jun 05 '20 at 05:47
1

yes, you have set dir before execution. for above question check this - https://unix.stackexchange.com/questions/37660/order-of-redirections – Srinivas Jun 05 '20 at 05:48
what this --- cut -d ":" -f2 | -- do here ? – BdEngineer Jun 05 '20 at 08:46
1

cut will split the fields, -d will be delimiter & -f will be the field in reverse order – Srinivas Jun 05 '20 at 08:53
, one more doubt what is this awk '$1=$1' ...what does mean here ? – BdEngineer Jun 05 '20 at 09:45
1

trimming additional spaces something like - echo " Running " | awk '$1=$1' – Srinivas Jun 05 '20 at 09:56
any advice how to solve this in spark ... https://stackoverflow.com/questions/62933135/dataframe-look-up-and-optimization – BdEngineer Jul 16 '20 at 10:59

score 1 · Answer 2 · answered Jun 05 '20 at 05:04

1

spark-submit is a async job, so when we submit the command you can get the application id by Calling SparkContext.applicationId. You can then check the status.

reference-https://issues.apache.org/jira/browse/SPARK-5439

if spark is deployed on Yarn then you can check the status using-

///To get application ID use yarn application -list
yarn application -status application_1459542433815_0002

There is another way they mentioned in this answer

answered Jun 05 '20 at 05:04

Som

6,193
1
11
22

how to get this applicationId inside the shell script ?..which invokes spark-submit – BdEngineer Jun 05 '20 at 05:50

how to capture the job status in shell script for spark-submit

2 Answers2