Bash/Shell | How to prioritize quote from IFS in read

Question

I'm working with a hand fill file and I am having issue to parse it. My file input file cannot be altered, and the language of my code can't change from bash script.

I made a simple example to make it easy for you ^^

var="hey","i'm","happy, like","you"
IFS="," read -r one two tree for five <<<"$var"
echo $one:$two:$tree:$for:$five

Now I think you already saw the problem here. I would like to get

hey:i'm:happy, like:you:

but I get

hey:i'm:happy: like:you

I need a way to tell the read that the " " are more important than the IFS. I have read about the eval command but I can't take that risk.

To end this is a directory file and the troublesome field is the description one, so it could have basically anything in it.

original file looking like that

"type","cn","uid","gid","gecos","description","timestamp","disabled"
"type","cn","uid","gid","gecos","description","timestamp","disabled"
"type","cn","uid","gid","gecos","description","timestamp","disabled"

Edit #1

I will give a better exemple; the one I use above is too simple and @StefanHegny found it cause another error.

while read -r ldapLine
    do
            IFS=',' read -r objectClass dumy1 uidNumber gidNumber username description modifyTimestamp nsAccountLock gecos homeDirectory loginShell createTimestamp dumy2 <<<"$ldapLine"

            isANetuser=0

            while IFS=":" read -r -a class
            do
                    for i in "${class[@]}"
                    do
                            if [ "$i" == "account" ]
                            then
                                    isANetuser=1
                                    break
                            fi
                    done
            done <<< $objectClass

            if [ $isANetuser == 0 ]
            then
                    continue
            fi

            #MORE STUFF APPEND#

    done < file.csv

So this is a small part of the code but it should explain what I do. The file.csv is a lot of lines like this:

"top:shadowAccount:account:posixAccount","Jdupon","12345","6789","Jdupon","Jean Mark, Dupon","20140511083750Z","","Jean Mark, Dupon","/home/user/Jdupon","/bin/ksh","20120512083750Z","",""

Answer 1

If the various bash versions you will use are all more recent than v3.0, when regexes and BASH_REMATCH were introduced, you could use something like the following function: [Note 1]

each_field () {
    local v=,$1;
    while [[ $v =~ ^,(([^\",]*)|\"[^\"]*\") ]]; do
        printf "%s\n" "${BASH_REMATCH[2]:-${BASH_REMATCH[1]:1:-1}}";
        v=${v:${#BASH_REMATCH[0]}};
    done
}

It's argument is a single line (remember to quote it!) and it prints each comma-separated field on a separate line. As written, it assumes that no field has an enclosed newline; that's legal in CSV, but it makes dividing the file into lines a lot more complicated. If you actually needed to deal with that scenario, you could change the \n in the printf statement to a \0 and then use something like xargs -0 to process the output. (Or you could insert whatever processing you need to do to the field in place of the printf statement.)

It goes to some trouble to dequote quoted fields without modifying unquoted fields. However, it will fail on fields with embedded double quotes. That's fixable, if necessary. [Note 2]

Here's a sample, in case that wasn't obvious:

while IFS= read -r line; do
  each_field "$line"
  printf "%s\n" "-----"
done <<EOF
type,cn,uid,gid,gecos,"description",timestamp,disabled
"top:shadowAccount:account:posixAccount","Jdupon","12345","6789","Jdupon","Jean Mark, Dupon","20140511083750Z","","Jean Mark, Dupon","/home/user/Jdupon","/bin/ksh","20120512083750Z","",""

EOF

Output:

type
cn
uid
gid
gecos
description
timestamp
disabled
-----
top:shadowAccount:account:posixAccount
Jdupon
12345
6789
Jdupon
Jean Mark, Dupon
20140511083750Z

Jean Mark, Dupon
/home/user/Jdupon
/bin/ksh
20120512083750Z


-----

---

Notes:

1. I'm not saying you should use this function. You should use a CSV parser, or a language which includes a good CSV parsing library, like python. But I believe this bash function will work, albeit slowly, on correctly-formatted CSV files of a certain common CSV dialect.

2. Here's a version which handles doubled quotes inside quoted fields, which is the classic CSV syntax for interior quotes:

   each_field () { 
       local v=,$1;
       while [[ $v =~ ^,(([^\",]*)|\"(([^\"]|\"\")*)\") ]]; do
           echo "${BASH_REMATCH[2]:-${BASH_REMATCH[3]//\"\"/\"}}";
           v=${v:${#BASH_REMATCH[0]}};
       done
   }
   

Answer 2

My suggestion, as in some previous answers (see below), is to switch the separator to | (and use IFS="|" instead):

sed -r 's/,([^,"]*|"[^"]*")/|\1/g'

This requires a sed that has extended regular expressions (-r) however.

Should I use AWK or SED to remove commas between quotation marks from a CSV file? (BASH)

Is it possible to write a regular expression that matches a particular pattern and then does a replace with a part of the pattern