Remove double quotes from all fields in an array in bash

Question

This array exists in bash script under a minimal linux/posix environment. All the fields in my string array are surrounded by double quotes. I am seeking an elegant solution to removing the double quote character at the beginning and end of each field, as there may be double quotes within the field that should not be removed.

The array is single dimensional, and contains fields as follows:

"This is a value, in this element"
"This is also a "value" but has double quotes"
"0X:41:DE:AD:BE:EF; -- EXIT --"

The desired field values after the operation would be as follows:

This is a value, in this element
This is also a "value" but has double quotes
0X:41:DE:AD:BE:EF; -- EXIT --

Currently I've tried the following without success:

fields=`sed -e 's/^"//' -e 's/"$//' <<<"${fields[@]}"

PLease see this answer: https://stackoverflow.com/a/26314887/2805824 — Gaurav Pathak, Apr 12 '23 at 13:14
@GauravPathak that would remove the quotes inside the field too. — Ed Morton, Apr 12 '23 at 13:32
Yes that’s right, I noticed only the heading—> remove double quotes from all fields — Gaurav Pathak, Apr 12 '23 at 15:23

score 8 · Accepted Answer · answered Apr 12 '23 at 13:25

8

Assuming that “in Bash” means “without external processes”, you can apply the usual Bash expansions / transformations to each array element. This yields the desired output, for example:

fields=('"This is a value, in this element"'
        '"This is also a "value" but has double quotes"'
        '"0X:41:DE:AD:BE:EF; -- EXIT --"')
        
fields=("${fields[@]/#\"}")  # remove leading quotes
fields=("${fields[@]/%\"}")  # remove trailing quotes

printf '%s\n' "${fields[@]}"

answered Apr 12 '23 at 13:25

Andrej Podzimek

2,409
9
12

Nice. You beat me to it while I was trying to figure a way to do the edits this way in one pass, but globs aren't regexes, lol - fun exercise, though. – Paul Hodges Apr 12 '23 at 14:26
1

The “in one pass” thing would be feasible if it weren’t for the requirement to keep quotes other than leading and trailing. `fields=("${fields[@]//\"}")` would remove all quotes in such case. – Andrej Podzimek Apr 13 '23 at 12:21
Yep. The only problem here is that globs are always anchored at both ends. :) – Paul Hodges Apr 13 '23 at 13:23

Ed Morton · Answer 2 · 2023-04-12T14:47:03.183

If this is the same array you asked for help populating recently, you could remove those quotes where it's being populated rather than in a separate stage afterwards. The awk script I'm using there is specifically adding double quotes to it's output because that's what you show you wanted in your question - obviously it doesn't need to do that and it could trivially remove the remaining quotes.

Here's one way to enhance the answer to your previous question to not populate the fields[] array with surrounding quotes:

$ cat tst.sh
#!/usr/bin/env bash

while IFS= read -d '' -r record; do
    readarray -d $'\n' -t fields <<< "$record"
    printf '%s\n' "${fields[@]}"
done < <( awk -v ORS='\0' '{gsub(/","/,"\n"); gsub(/^"|"$/,"")} 1' "${@:--}" )

$ ./tst.sh file
AC XA, S.A.
City of Commerce
00
0A348E541E6F5C258A12A5674AEF25F28BA7DCFAECEECC4EE63B71B361606AC3
Included
Included
Included
Not Included
09/30/2003
09/30/2037

SZ=City of Commerce; FE=http://website.org; O=AC XA SA CIF A39201827; C=QR
--BINARY BLOB--maXJtYSBTQSBDSUYgQTgyNzQzMjg3MSMwIQYDVQQLExpodHRwOi8vd3d3LmNoYW1iZXJzaWduLm9yZzEiMCAGA1UEAxMZQ2hhbWJlcnMgb2YgQ29tbWVyY2UgUm9vdDAeFw0wMzA5MzAUIC0gMjAxNjESMBAGA1UEBRMJQTgyNzQzMjg3MRgwFgYDVQRhDA9WQVRFUy1BODI3NDMyODcxGzAZBgNVBAoMEkFDIENBTUVSRklSTUEgUy5BLjEnMCUGA1UEAwweR0xPQkFMIENIQU1CRVJTSUdOIFJPT1QgLSAyMDE2MB4XDTE2MDQxNDA3NTAwNloXDTQwMDQwODA3NTAwNlowggEIMQswCQYDVQQGEwJFUzEPMA0GA1UECAwGTUFEUklEMQ8wDQYDVQQHDAZNQURSSUQxOjA4BgNVBAsMMXNlZSBjdXJyZW50IGFkZHJlc3MgYXQgd3d3LmNhbWVyZmlybWEuY29tL2FkZHJlc3MxJzAlBgNVBAsMHkdMT0JBTCBDSEFNQkVSU0lHTiBST09UIC0gMjAxNjESMBAGA1UEBRMJQTgyNzQzMjg3MRgwFgYDVQRhDA9WQVRFUy1BODI3NDMyODcxGzAZBgNVBAoMEkFDIENBTUVSRklSTUEgUy5BLjEnMCUGA1UEAwweR0xPQkFMIENIQU1CRVJTSUdOIFJPT1QgLSAyMDE2MIICIjANBgkqhkiG9w0BAQEF--END--
SZ1qSRW
Email
AFX Client;Email
AFX Method;AFX Method;Entry Point;AFX Email;Time Stamping
No Entry

We're adding readarray now as we can't populate fields[] directly while reading in this case since an empty field would result in back-to-back \ns and when IFS is set to a white-space character such as \n, read uses contiguous appearances of that character as a single separator and so empty fields would be skipped.

score 0 · Answer 3 · answered May 02 '23 at 13:49

maybe mapfile plus sed:

fields=('"This is a value, in this element"'
        '"This is also a "value" but has double quotes"'
        '"0X:41:DE:AD:BE:EF; -- EXIT --"')

mapfile -t newfields < <(printf "%s\n" "${fields[@]}" | sed 's/^"//;s/"$//')

printf "%s\n" "${newfields[@]}"

This is a value, in this element
This is also a "value" but has double quotes
0X:41:DE:AD:BE:EF; -- EXIT --

score 0 · Answer 4 · answered May 03 '23 at 14:47

#!/bin/bash

fields=('"This is a value, in this element"'
        '"This is also a "value" but has double quotes"'
        '"0X:41:DE:AD:BE:EF; -- EXIT --"')

echo "--> before"
declare -p fields

for i in "${!fields[@]}"
do
    value="${fields[i]/\"/}"
    value="${value%"${value##*[!\"]}"}"
    fields[i]=$value
done

echo -e "\n--> after"
declare -p fields

Output

--> before
declare -a fields=([0]="\"This is a value, in this element\"" [1]="\"This is also a \"value\" but has double quotes\"" [2]="\"0X:41:DE:AD:BE:EF; -- EXIT --\"")

--> after
declare -a fields=([0]="This is a value, in this element" [1]="This is also a \"value\" but has double quotes" [2]="0X:41:DE:AD:BE:EF; -- EXIT --")

Remove double quotes from all fields in an array in bash

4 Answers4