0

I have set of lines in a text file . Each set consists of 4 lines which describes about a Id,attemptID,SparkUser,starttimeEpoch and attemptID in a single file. I would like to parse and assign every FOUR lines to four variable on single shot and move on to the next set of content. I have a below script but it is not working as expected.

I am used a while loop which read every line and grep a string and assigns variable to CURL command ( A API call to spark ) . In the curl command i m having two variables which can be read only in two consective lines. If I use While loop , chances are it can assign only one variable at a time and the CURL command would expect to have two variables assigned to it , so the CURL is failing here. 

while read line
do
attempt_id=`echo $line | grep -w attemptId | awk '{print $6}' | cut -d'"' -f2`
id=`echo $line| grep id | awk '{print $3}' | cut -d'"' -f2`
user=`echo $line | grep "sparkUser" | awk '{print $6}' | cut -d'"' -f2`
start_time=`echo $line |  grep "startTimeEpoch" | awk '{print $9}' | cut -d'"' -f2 | cut -d',' -f1 | cut -c 1-10`

if [[ ! -z "$attempt_id" ]]
then
id=`echo $line| grep id | awk '{print $3}' | cut -d'"' -f2`
user=`echo $line | grep "sparkUser" | awk '{print $6}' | cut -d'"' -f2`
# the below variables would convert Epoch Milliseconds into Human readable time format:
#
start_time=`echo $line |  grep "startTimeEpoch" | awk '{print $9}' | cut -d'"' -f2 | cut -d',' -f1 | cut -c 1-10`
 time=`date -d @$start_time`
 a=`curl -k  -u $USERNAME:$PASS -H "Accept: application/json" -X GET -s -k http://XXXXXXXXXX:18080/history/$id/$attempt_id/environment/ |  xargs | grep -Po "<tr>\K(.*?)</tr>" | sed "s/..tr.//g" | grep spark.submit.deployMode | grep -ic client`


done < wi

Sample File:

    "attemptId" : "1",
    "sparkUser" : "XXXXX",
    "startTimeEpoch" : 1559782915432
  "id" : "application_1558744311646_179708",
    "attemptId" : "1",
    "sparkUser" : "yyyyy",
    "startTimeEpoch" : 1559782769130
masteraravind
  • 45
  • 1
  • 1
  • 4
  • 3
    Do not parse json or XML with bash or shell utilities, instead use a proper tool for the job like `jq` and `xmlstartlet`. See, e.g. [Parsing JSON with Unix tools](https://stackoverflow.com/questions/1955505/parsing-json-with-unix-tools) and – David C. Rankin Jun 06 '19 at 01:52
  • 1
    Possible duplicate of [Parsing JSON with Unix tools](https://stackoverflow.com/q/1955505/608639), [Read JSON data in a shell script](https://stackoverflow.com/q/20488315/608639), etc. – jww Jun 06 '19 at 02:59
  • 1
    As an aside, anything which looks like `grep 'x' | awk '{ y }'` can be usefully refactored to `awk '/x/ { y }'`. See also [useless use of `grep`.](http://www.iki.fi/era/unix/award.html#grep) You can easily get rid of the `cut` too; `id=$(echo "$line" | awk '/id/ { split($3, x, /"/); print x[2] }')` – tripleee Jun 06 '19 at 03:57

0 Answers0