Shell script if statement variable comparison issues

Question

I have the following shell script to append a header line to a batch of .csv files. I need to apply a different header txt file to each .csv based on the number of columns in the input csv. The script is working in that it is applying the header files and outputting the new csv, but every file is getting the 'else' header--including the csv's that evaluate to '91' with the csvgrep -n $f | wc -l command.

#!/bin/sh

dir="/Users/XXX/xlsxtocsv"

#move to the csvraw output directory
cd csvraw/

#append the header row needed for DB import
for f in *.csv; do 
    if [[ "csvgrep -n $f | wc -l" == 91 ]]; then
       cat $dir/svcheader3col.txt $f >> "$dir/csvhead/${f%}header.csv"
    else
       cat $dir/svcheader.txt $f >> "$dir/csvhead/${f%}header.csv"
    fi

done

And when I run the csvgrep command manually on the csv files I get:

username:csvraw username$ csvgrep -n file4col.csv | wc -l
      92
username:csvraw username$ csvgrep -n file3col.csv | wc -l
      91

Why is the "file3col.csv" file not evaluating to 91 in the if statement, but is when I run it manually?

It's comparing the string `csvgrep -n $f | wc -l`, not the output of the command `csvgrep -n $f | wc -l`. — Charles Duffy, Oct 02 '18 at 16:36
BTW, consider using `bash -x yourscript` to run a script logging each command it invokes -- if you actually *see* what the test is doing in practice, that makes this kind of issue much easier to debug. — Charles Duffy, Oct 02 '18 at 16:42

score 1 · Accepted Answer · edited Oct 02 '18 at 17:06

1

You aren't actually running the csvgrep command. Use a command substitution such as $(...) to do so.

#!/bin/bash
: "${dir?dir must be defined}"
for f in *.csv; do 
    if [[ "$(csvgrep -n "$f" | wc -l | tr -d '[:space:]')" = 91 ]]; then
       cat -- "$dir/svcheader3col.txt" "$f" 
    else
       cat -- "$dir/svcheader.txt" "$f"
    fi >> "$dir/csvhead/${f%.csv}header.csv"
done

edited Oct 02 '18 at 17:06

jamierob

127
2
7

answered Oct 02 '18 at 16:37

Charles Duffy

280,126
43
390
441

This is working much better and I understand what I was doing wrong. 'bash -x' outputs the following, which seems like it's not getting the 'wc -l' part correct. Is the whitespace in front of '92' and '91' messing things up as well? '+ for f in '*.csv' '++ csvgrep -n file4col.csv '++ wc -l '+ [[ 92 == 91 ]] '+ cat -- /Users/xlsxtocsv/svcheader.txt file4col.csv '+ for f in '*.csv' '++ csvgrep -n file3col.csv '++ wc -l '+ [[ 91 == 91 ]] '+ cat -- /Users/xlsxtocsv/svcheader.txt file3col.csv – jamierob Oct 02 '18 at 16:53
Could you backtick-escape that? Formatted as it is there, the `*`s are being treated as formatting. (That said, a longer log is probably easier to post as a [gist](https://gist.github.com/) so multiline formatting sticks). That said, it looks to me like it's doing what it should from that log. (Note that the `%.csv` means this code is writing output to `CNY04-3colheader.csv`, not `CNY04-3col.csvheader.csv` as the original would have done). – Charles Duffy Oct 02 '18 at 16:54
What whitespace? `[[ 92 == 91 ]]` is exactly what it *should* look like. Whitespace is mandatory in test operations -- keep in mind that `[` is a command, not shell syntax, and commands are separated from their arguments with whitespace. (`[[ ]]` is actually syntax, but still, the whitespace is mandatory). If there were whitespace *in your data*, then the `bash -x` logs would show it as something like `[[ ' 91' == 92 ]]`. – Charles Duffy Oct 02 '18 at 16:57
(BTW, `=` rather than `==` in this answer was deliberate: `==` is a nonportable extension, whereas `=` as a string comparison operator works even in baseline POSIX `test`/`[`; see http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html) – Charles Duffy Oct 02 '18 at 17:00
Here's the output form bash -x with the whitespace. I'm still unsure how to get '91 to evaluate correctly. https://gist.github.com/jamierob/d18f3d751893a416b6acae919cf9b3ea – jamierob Oct 02 '18 at 17:00
Ahh. Easiest thing to do is probably to strip the whitespace. It's kinda' ugly/slow, but `wc -l | tr -d '[:space:]'` should do the trick. – Charles Duffy Oct 02 '18 at 17:02
Charles - The extra space was it and now it's working! Thank you! – jamierob Oct 02 '18 at 17:05
BTW, another approach would be `[[ "$(csvgrep -n "$f" | wc -l)" =~ ^[[:space:]]*91[[:space:]]*$ ]]`. – Charles Duffy Oct 02 '18 at 17:06

Shell script if statement variable comparison issues

1 Answers1