0

I am trying to execute the below code snippet to determine the presence of an entry in an extract. The problem is that the awk is not working properly in the script whereas just as a command is it working perfectly.

Let me know if anything else is required!

-bash-4.2$ cat checkSub.sh 
#!/bin/bash -x

path=`pwd`

while IFS= read -r line
do
    SUB="`echo $line | cut -d"," -f1`"
    VAL="`awk -F',' '$1 == "$SUB" {print 1}' ${path}/count.csv`"
    #VAL=`cat ${path}/count.csv | grep $SUB | wc -l`
    echo $VAL

     if [ $VAL == 1 ];then
            echo "$SUB has entry in the output sheet!"
     else
            echo "$SUB doesn't have an entry!"
     fi

done < ${path}/GTPL_SLA.dat

PFB output. I am thinking if it has anything to do with using print to assign value or something is wrong with the syntax itself

-bash-4.2$ ./checkSub.sh 
++ pwd
+ path=<designated path>
+ IFS=
+ read -r line
++ echo STD_GTPL_UNSETL_1_NY_DL,01:30:00
++ cut -d, -f1
+ SUB=STD_GTPL_UNSETL_1_NY_DL
++ awk -F, '$1 == "$SUB" {print 1}' <designated path>/count.csv
+ VAL=
+ echo

+ '[' -eq 1 ']'
./checkSub.sh: line 12: [: -eq: unary operator expected
+ echo 'STD_GTPL_UNSETL_1_NY_DL doesn'\''t have entry!'
STD_GTPL_UNSETL_1_NY_DL doesn't have an entry!
+ IFS=
+ read -r line
++ echo STD_GTPL_UNSETL_2_CA_DL,00:00:00
++ cut -d, -f1
+ SUB=STD_GTPL_UNSETL_2_CA_DL
++ awk -F, '$1 == "$SUB" {print 1}' <designated path>/count.csv
+ VAL=
+ echo

+ '[' -eq 1 ']'
./checkSub.sh: line 12: [: -eq: unary operator expected
+ echo 'STD_GTPL_UNSETL_2_CA_DL doesn'\''t have entry!'
STD_GTPL_UNSETL_2_CA_DL doesn't have entry!
+ IFS=
+ read -r line

As a standalone command, it works like a charm :

-bash-4.2$ awk -F',' '$1 == "STD_GTPL_UNSETL_1_NY_DL" {print 1}' count.csv
1
-bash-4.2$

In the end, I have opted for grep, but not a big fan of it, as it gives the desired result.

VAL=`cat ${path}/count.csv | grep $SUB | wc -l`

-bash-4.2$ ./checkSub.sh 
1
STD_GTPL_UNSETL_1_NY_DL has entry in the output sheet!
0
STD_GTPL_UNSETL_2_CA_DL doesn't have entry!
-bash-4.2$

GTPL_SLA.dat

-bash-4.2$ cat GTPL_SLA.dat 
STD_GTPL_UNSETL_1_NY_DL,01:30:00
STD_GTPL_UNSETL_2_CA_DL,00:00:00
-bash-4.2$

count.csv

-bash-4.2$ cat count.csv 
STD_GTPL_UNSETL_1_NY_DL,9/8/2021,2021-09-08~22:25:20,2021-09-08~22:28:21,Completed,898724
-bash-4.2$ 

Ed, adding the code after following suggestions from shellcheck.net

-bash-4.2$ cat checkSub.sh 
#!/bin/bash -x

#path=$(pwd)

while IFS= read -r line
do
        VAL=0
        SUB=$(echo "$line" | cut -d"," -f1)
        VAL=$(awk -F"," -v a="$SUB" '"$1" == a {print 1}' count.csv)
        #VAL=`cat ${path}/count.csv | grep $SUB | wc -l`
        echo "$VAL"

        if [ "$VAL" -eq 1 ];then
                echo "$SUB has entry in the output sheet!"
        else
                echo "$SUB doesn't have entry!"
        fi

done < GTPL_SLA.dat
vkeeWorks
  • 89
  • 7
  • 1
    Read the question I closed this as a dup of and copy/paste your script into http://shellcheck.net and fix the issues it tells you about then post a new question if you still have one. See also [why-is-using-a-shell-loop-to-process-text-considered-bad-practice](https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice) as you shouldn't be using a shell read loop for this anyway though, just a single call to awk. – Ed Morton Sep 10 '21 at 12:52
  • When comparing a command used in a script to the same command used on the command line to look for differences it's important to actually use **exactly** the same command. `awk -F',' '$1 == "STD_GTPL_UNSETL_1_NY_DL" {print 1}'` is not the same as `awk -F',' '$1 == "$SUB" {print 1}'`. Try changing your command-line script to `SUB=STD_GTPL_UNSETL_1_NY_DL; awk -F',' '$1 == "$SUB" {print 1}'` as THEN it'll be the same as in your script and you'll see both fail the same way as the problem has nothing to do with awk being called in a script. – Ed Morton Sep 10 '21 at 12:55
  • The data set to loop over varies and at max, it is about 120-150, hence execution time is not that much, thus we use `while` as we do want to check each and every line carefully before preparing the report. – vkeeWorks Sep 10 '21 at 13:02
  • is there any other way that you can suggest which works better than `while` `read` ?. Also, you mentioned that command should exactly be same, but then what's the issue here exactly then cause variable `SUB` is having the required value and should be able to render it while it is being used to compare with another variable, isn't it?! – vkeeWorks Sep 10 '21 at 13:03
  • None of that matters to the problem you're asking about. The problem you asked about is a duplicate of the question I closed yours as a dup of so read that. Yes, there is a far better way to do what you want. Ask a new question about improving the robustness, efficiency, and portability of your script once you've created a new script with your current problem and the issues that shellcheck will tell you about fixed so we can help you with that. – Ed Morton Sep 10 '21 at 13:03
  • ok, you want me to just post the question again asking "how to improve the performance of the said script"? – vkeeWorks Sep 10 '21 at 13:08
  • Definitely not. Please re-read my comment. **Fix** your script with the information I just gave you and then post a new, different question that has nothing to do with the problem you asked about here (obviously since you'll then no longer have that problem) and is instead about improving the script you **will have** after fixing the current issues. – Ed Morton Sep 10 '21 at 13:10
  • Please check once more. I have mentioned that i have instead used `grep` to get the desired result and just wanted to know why isn't `awk` part of the script working. Not sure what is to be fixed exactly! Are you referring to the usage of `read` ? – vkeeWorks Sep 10 '21 at 13:20
  • Have you read the question I closed yours as a dup of? It tells you exactly what is wrong with your awk script which is why I provided that reference for you. – Ed Morton Sep 10 '21 at 13:21
  • Actually i did read your answer, which was clear and precise, and to prove that i did the following changes and it is working with `awk` too now : `awk -v a="$SUB" '$1 == $a {print 1}' ${path}/count.csv` – vkeeWorks Sep 10 '21 at 13:52
  • You sure that's working? I'd have expected you to need `$1 == a` instead of `$1 == $a` but I may be misunderstanding what your `SUB` variable contains. Anyway, glad that helped and as I mentioned, once you've fixed that and the issues that shellcheck tells you about you can, if you want, ask a new question including that updated script about how to improve it's performance, robustness, and portability. – Ed Morton Sep 10 '21 at 13:55
  • You're right, it is still not working. Tried with `$1 == a` as well, but still issue persists. For `awk -v a=STD_GTPL_UNSETL_1_NY_DL '$1 == $a {print 1}'`, VAL=1 as it should but for `awk -v a=STD_GTPL_UNSETL_2_CA_DL '$1 == $a {print 1}'` VAL should be 0 but it is getting assigned with 1. I am reverting the value to 0 for VAL before the next iteration hence i can see the values change from 1 (correct) to 0 to 1 (incorrect). I also tried `awk -v a=$SUB '$1 == a {print 1}' ${path}/count.csv` but that is not assigning anything!! – vkeeWorks Sep 10 '21 at 14:10
  • Just from the snippet of code in your comment I can see that you haven't copy/pasted your code into http://shellcheck.net and fixed the issues it'll tell you about as I've suggested you do multiple times now. We solved the problem you asked about, please do what else I suggested and then simply ask a new question if you have a new problem - whether that's about improving working code or helping you fix some new bug in your code doesn't matter, whatever it is you need help with, just ask a new question after upading your code such that shellcheck runs cleanly. – Ed Morton Sep 10 '21 at 14:13
  • I did copy paste the exact code at the website and followed the instructions to use latest syntax, add double quotes wherever mentioned, etc. and then it mentioned that `$shellcheck myscript No issues detected!`. After that also when i executed my script, it isn't working. – vkeeWorks Sep 10 '21 at 14:36
  • Then what you posted in your comment (`awk -v a=$SUB '$1 == a {print 1}' ${path}/count.csv`) is not the same as the code you're now executing because shellcheck would not let those unquoted variables `$SUB` and `${path}` go unreported and if that's different from the code you're really executing then idk what else might be different. I wont be able to continue trying to help you in comments here, please just ask a new question (not just a copy of this one of course as you now have different code and a different problem). – Ed Morton Sep 10 '21 at 14:39
  • I did follow the instructions and added quotes as suggested : `VAL=$(awk -F"," -v a="$SUB" '"$1" == a {print 1}' count.csv)`. I removed `${path}` parameter since the files were present in same test directory as the script – vkeeWorks Sep 10 '21 at 14:42
  • @Ed, I added the code at the end after following instructions from the site. – vkeeWorks Sep 10 '21 at 14:45
  • 1
    last time - ask a new question. – Ed Morton Sep 10 '21 at 14:47

0 Answers0