0

So I'm trying to write a script that will take any given CSV file, with a non-fixed number of rows/columns, and parse each field as an input for another part of the script.

Script should take each field, and store it in a variable named, let's say f{var} where {var} is the number variable it is. These variables will later need to be parsed to recognize IP addresses from IP addresses+CIDR (x.x.x.x/yy). However, before even getting to figuring out how to do variable amounts of variables, I am having a few issues (pretty new to scripting).

I've been experimenting with some of the solutions found on this and other sites, but the behavior I'm observing when testing doesn't seem to match what's described. For example, grep -o "...$" (to pull the last 3 characters) seems to return only 2 characters not 3 in some cases. Both lines in my test csv are 4 columns of IP addresses.

I ran something like this

input="/home/admin/parsetest.cvs" 
while IFS=',' read -r f1 f2 f3 f4
do
  echo "$f1 $f2 $f3 $f4"
done < "$input"

But it only produced a single line, rather than echo'ing both lines to STDOUT. Looking at it with -xv on, it looks like it reads the first line, echo's it, checks IFS, reads again, and then terminates. I'm fairly certain I'm doing something wrong, but I'm not sure what. 

[Expert@LabFW01:0]# ./csvparse.sh 
#!/bin/bash -xv 
echo "IFS delimiter set to ," 
+ echo 'IFS delimiter set to ,' 
IFS delimiter set to , 

input="/home/admin/parsetest.csv" 
+ input=/home/admin/parsetest.csv 
while IFS=',' read -r f1 f2 f3 f4 
do 
    echo "$f1 $f2 $f3 $f4" 
done < $input 
+ IFS=, + read -r f1 f2 f3 f4 ' 
echo '192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 
+ IFS=, + read -r f1 f2 f3 f4
John Stefansson
  • 1
  • 1
  • 1
    `read -a` is your friend. – Charles Duffy Dec 07 '15 at 19:46
  • 1
    ...however, if your goal is to build a shell library for CSV parsing, one already exists, and it's a lot more complex than what you've got here. The CSV format allows literal commas within contents; requires quote handling; allows those quotes to be escaped... it's not nearly so trivial. – Charles Duffy Dec 07 '15 at 19:47
  • As for your second `read` failing, that depends on the input (unless you have something much more interesting than an `echo` inside your loop). Inlining input, ie. `< – Charles Duffy Dec 07 '15 at 19:48
  • 1
    To quote the "csv" factoid from http://wooledge.org/~greybot/meta/csv: A csv file contains "Comma Separated Values". It represents records as lines and fields delimited by commas (though the delimiter can vary). Very simple CSV files can be parsed using a `while IFS=, read -a fields` loop. For more complete support, see , or the csvtool / csvkit commands. – Charles Duffy Dec 07 '15 at 19:50
  • Backing up, though: "Something like this" is not good enough **unless the example actually reproduces your problem in practice**. Test that a reproducer really does have the same bug your original code had before you post it! – Charles Duffy Dec 07 '15 at 19:51
  • ...I'd place my $.02 bet that you're actually running `ssh` or something else that consumes stdin inside your loop, making this a duplicate of http://stackoverflow.com/questions/9393038/ssh-breaks-out-of-while-loop-in-bash – Charles Duffy Dec 07 '15 at 19:53
  • Your symptoms make me guess that your input is from Windows and contains `\r\n` line endings? That could explain both issues. – Jeff Y Dec 07 '15 at 19:57
  • 1
    When you use `od -cx` you can check the last character. Is it a newline? – Walter A Dec 07 '15 at 20:35
  • You're right that I was using SSH, but even when I'm not, I still get pretty much identical output. `[Expert@LabFW01:0]# ./csvparse.sh #!/bin/bash -xv echo "IFS delimiter set to ," + echo 'IFS delimiter set to ,' IFS delimiter set to , input="/home/admin/parsetest.csv" + input=/home/admin/parsetest.csv while IFS=',' read -r f1 f2 f3 f4 do echo "$f1 $f2 $f3 $f4" done < $input + IFS=, + read -r f1 f2 f3 f4 ' echo '192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 + IFS=, + read -r f1 f2 f3 f4` Sorry about the formatting how to fix – John Stefansson Dec 08 '15 at 21:12
  • @Jeff: I ran dos2unix against both files, but yes, it was originally from windows. – John Stefansson Dec 09 '15 at 16:41
  • You can fix the format by editing your question (adding it and use `{}`). – Walter A Dec 09 '15 at 22:17
  • @WalterA Alright, updated the debug output into the main question. – John Stefansson Dec 14 '15 at 15:32
  • The output of the suggested command `od -cx /home/admin/parsetest.csv` would have helped. – Armali Sep 01 '17 at 12:48

0 Answers0