Loop over tuples in bash, separated by a newline

Question

I have a file that looks like this:

14757,file_one
14756,file_two
14755,file_three

I want to loop over each line and refer to tuple components by separate variables. For example, when iterating over the first line, $1 would be 14757 and $2 would have a value of file_one.

I try to achieve this with:

for i in $(cat files.txt); do IFS=","; set -- $i; echo $1 and $2; done

However, it loops over each word, and the result is not what I expect:

14757 and
file_one
14756 and
file_two
14755 and
file_three and

This is what I want:

14757 and file_one
14756 and file_two
14755 and file_three

I tried to adapt solutions posted in question Loop over tuples in bash? without success.

Answer 1

If that's what you feel you need, then

$: while IFS=$'",\n' read -a line; do set -- "${line[@]}"; shift; echo $1 and $2; done <tmp
14757 and file_one
14756 and file_two
14755 and file_three

I used the quotes as delimiters as well as the comma, which creates a leading empty field in cell 0, so I shift it off.

...but unless there is a compelling reason, just use the array.

$: while IFS=$'",\n' read -a fields; do echo "${fields[1]} and ${fields[2]}"; done <tmp
14757 and file_one
14756 and file_two
14755 and file_three

awk would be a lot more efficient, and notably faster if the result set is very big -

$: awk -F'[",]' '{print $2" and "$3}' tmp
14757 and file_one
14756 and file_two
14755 and file_three

or even sed -

$: sed 's/^"//; s/"$//; s/,/ and /;' tmp
14757 and file_one
14756 and file_two
14755 and file_three

This one is a little more direct and mechanical, but if you read regexes it's pretty easy to understand: trim the leading quote, trim the trailing quote, convert the comma. I could have used s/"//g, but I suspect the two anchored substitutions are faster than scanning the whole string since I know where the quotes are. It likely doesn't matter here, but worth mentioning for when you're processing a multi-GB file and you want to shave a little time.

If you actually did pipe your data through a tr and remove the quotes, then all these are a little simpler, as they don't have to deal with that anymore, and you don't ignore the first empty field.

$: while IFS=, read -a line; do set -- "${line[@]}"; echo $1 and $2; done <tmp
14757 and file_one
14756 and file_two
14755 and file_three

$: while IFS=, read -a fields; do echo "${fields[0]} and ${fields[1]}"; done <tmp
14757 and file_one
14756 and file_two
14755 and file_three

$: awk -F, '{print $1 " and " $2}' tmp
14757 and file_one
14756 and file_two
14755 and file_three

$: sed 's/,/ and /;' tmp
14757 and file_one
14756 and file_two
14755 and file_three

Answer 2

A variation on the while/read loop:

$ while IFS=, read -r arg1 arg2; do echo "${arg1} and ${arg2}"; done < files.txt
14757 and file_one
14756 and file_two
14755 and file_three

Answer 3

The naming of your variables, $1 and $2, is a give-away. You mean you have heard somewhere of a technology, called awk, but you have forgotten its name :-)

Let me show you an example:

awk -F " " '{print $1 " blabla " $2}' file.txt

Result:

14757 blabla file_one
14756 blabla file_two
14755 blabla file_three

For your information: the flag -F " " means that I use a space as a separator.

In case you have problems with double quotes in your output, you can simply remove them adding | tr -d "\"" at the end of your command, so you get:

awk -F " " '{print $1 " blabla " $2}' file.txt | tr -d "\""