Bash script to write a variable value that contains " and ' to a file

Question

I'm creating a bash script to write variable values to a csv file. But unable to write particular values that contain " and ' symbols in the line. For example, I have variables and values like this available into the shell:

username: my-name@abc.com

name: my-name

password: j50z54"#7b'y'/3l7H%7

If try to write them to a csv file like this:

  echo -e "username,password,name" >> abc.csv
  echo -e "$username,$password,$name" >> abc.csv

I get an error:

line 2: unexpected EOF while looking for matching `"'
line 3: syntax error: unexpected end of file

Thre problem occurs due to the presence of double quotes and single quotes in the value.

Method 1:

Store the variable value as follows:

• escape all double quotes and single quotes by prepending them with a backslash \
• enclose the entire variable value within double quotes.

For example, an actual value of password j50z54"#7b'y'/3l7H%7 should be saved as "j50z54\"#7b\'y\'/3l7H%7"

Then if I run the following script:

  echo -e "username,password,name" >> abc.csv
  echo -e "$username,$password,$name" >> abc.csv
  sed -i 's/\\//g' abc.csv

I get the proper csv file:

username,password,name
my-name@abc.com,j50z54"#7b'y'/3l7H%7,my-name

BUT, the problem is that it is not possible to modify the variable value ( at its source ) as in the above-mentioned format.

Method 2:

I tried a metho as shown here, the script:

username=my-name@abc.com
name=my-name
cat >> abc.csv << \EOF
Username,Password,Name
$username,j70z38"#7k'y'/3l7H%9,$name
EOF

gives output file abc.csv:

Username,Password,Name
$username,j70z38"#7k'y'/3l7H%9,$name

(please ignore the fact that password value is directly provided here for just testing purposes)

In this method, variable names like $username and $name are printed instead of their values.

Question:

Is there any other better way/script to write the variable value properly to a CSV file?

ADDITIONAL INFORMATION:

This script is used for automation inside a bash task of the Azure DevOps pipeline. Also, the password is saved in a variable group there and is available inside the pipeline as $(password).

Answer 1

Piecing together the clues, I think your situation is something along these lines:

• the script receives the username, password, and name from some external source (arguments, interactive input, file ...) and stores them in variables.

• you want to write a file containing CSV representations of those values

• your first attempt, with echo, writes the values as received, but your (separate) CSV reader cannot read them back successfully afterward

• your "Method 1" describes a proof-of-concept variation in which the username, password, and name are set explicitly in the script, instead of being read from an external source as they ultimately need to be.

• your "Method 2" is presumably another proof-of-concept variation

However, I do not accept this:

unable to write particular values that contain " and ' symbols in the line.

The example presented to support that claim in fact emits such values flawlessly, and, separately, the error message you present cannot have come from those lines. My best guess is that the error comes from a CSV reader, in which case it reflects not that you failed to write the data, but that the resulting file was not valid CSV (as far as that reader was concerned). Do note, however, that if the data contained any literal backslashes (\) then echo -e would screw that up. If you want to use echo (which you could) then drop the -e.

Additionally, the file resulting from your method 1 would be rejected by some CSV readers on account of containing a double quote in an unquoted field, and it would be rejected by most CSV readers if the double quote appeared at the beginning of the field instead of in the middle. It may be what you expected, but it is not well characterized as a "proper" csv file by most standards.

The issue you seem to be struggling with is that CSV requires special handling for some characters. You cannot just dump raw character data into a CSV file and expect readers to understand it. You need to format the data appropriately for CSV. This has nothing specifically to do with Bash.

Note also that there is a pretty big diversity of formats that fall under the "CSV" umbrella. Not all of them even use commas as field delimiters. If you are targeting a specific CSV dialect then you will need to format your output as that dialect requires. That will surely include some form of quoting for fields that include the field delimiter, and very likely some form of quoting for fields that include the quoting character(s). If your dialect allows fields that contain the record delimiter, then it will also require those fields to be quoted.

If you're not sure what your CSV dialect requires, or if you're flexible about that, then know that CSV dialects predominantly use the double-quote character (") for quoting, and generally they support quoting only on a whole-field basis. And typically, the designated means for escaping a double quote inside a quoted field is to double it. Some CSV dialects expect all fields to be quoted, and that's compatible with most dialects, so unless you know that you need to do differently, that's what I would recommend.

Example:

#!/bin/bash

# Input the data
read -r -p 'username: ' username
read -r -p 'name: '     name
read -r -p 'password: ' password

# Double all quotation marks (") in the field values
username_csv=${username//\"/\"\"}
name_csv=${name//\"/\"\"}
password_csv=${password//\"/\"\"}

# Write the output file with quotes and field delimiters
cat >output.csv <<EOF
Username,Password,Name
"$username_csv","$password_csv","$name_csv"
EOF

Demo

$ bash example.sh
username: my-name@abc.com
name: my-name
password: j50z54"#7b'y'/3l7H%7
$
$ cat output.csv
Username,Password,Name
"my-name@abc.com","j50z54""#7b'y'/3l7H%7","my-name"