Awk output not consistent

Question

I have created a bash script that is suppose to either append or overwrite some existing environment variables in /etc/environment. The script looks like this:

#!/bin/bash
declare -a array=("http_proxy=http://username:password@proxy:8080"
                  "https_proxy=http://username:password@proxy:8080"
                  "ftp_proxy=ftp://username:password@proxy:8080"
                  "no_proxy=\"localhost, 127.0.0.1\""
                 )

for i in "${array[@]}"; do
        awk -v var="${i%%=*}" '$0 !~ var' /etc/environment > /etc/environment
        echo "$i" >> /etc/environment
done

Given the following input (/etc/environment):

http_proxy=http://otheruser:password@otherproxy:8080
https_proxy=http://username:password@proxy:8080
some_env_varible=some_value

I expect the following output:

http_proxy=http://username:password@proxy:8080
https_proxy=http://username:password@proxy:8080
ftp_proxy=ftp://username:password@proxy:8080
no_proxy="localhost, 127.0.0.1"
some_env_variable=some_value

However i get the following output:

http_proxy=http://username:password@proxy:8080
https_proxy=http://username:password@proxy:8080
ftp_proxy=ftp://username:password@proxy:8080
no_proxy="localhost, 127.0.0.1"

Running the awk command outside of the script seems to work fine, returning only the lines that do not match a specific environment variable. I am puzzled to why this is not working inside this script.

Update

As per request:

$ sudo cat /etc/environment
http_proxy=http://username:password@proxy:8080
https_proxy=http://username:password@proxy:8080
ftp_proxy=ftp://username:password@proxy:8080
no_proxy="localhost, 127.0.0.1"
$ i="http_proxy=http://username2:password2@proxy:8080"
$ sudo awk -v var="${i%%=*}" '$0 !~ var' /etc/environment
https_proxy=http://username:password@proxy:8080
ftp_proxy=ftp://username:password@proxy:8080
no_proxy="localhost, 127.0.0.1"

`awk ... /etc/environment > /etc/environment` is no good idea. — Cyrus, Jun 25 '19 at 08:39
Please post the *awk command outside of the script*, I doubt it could *work fine* (as pointed out by @Cyrus). — simlev, Jun 25 '19 at 08:42
To check whether the variable exists: `if ! grep -q "${i%%=*}" file; then echo "$i" >> /etc/environment; fi`. — simlev, Jun 25 '19 at 08:58
@simlev check the update. Notice that the purpose is also to overwrite existing variables, and not just append if it doesn't exist. — sarnikowski, Jun 25 '19 at 09:03

kvantour · Accepted Answer · 2019-06-25T10:30:24.087

It looks a bit cumbersome what you are trying to achieve, you might want to consider any of the following options:

updating the original script:

The awk line you write tries to print everything which is not in the file /etc/environment. This is exactly what grep -v does :

for i in "${array[@]}"; do
   grep -v "${i%%=*}" /etc/environment > /etc/environment.new
   echo "$i" >> /etc/environment.new
   mv /etc/environment.new /etc/environment
done

Exploit the power of grep:

Now that we use grep, you should be aware of the flag

-f FILE, --file=FILE Obtain patterns from FILE, one per line. The empty file contains zero patterns, and therefore matches nothing.

So, making use of process-substitution (<(command)), you can now do something like:

grep -v -f <(printf "%s\n" "${array[@]}" | cut -d= -f1) -F /etc/environment > /etc/environment.new
printf "%s\n" "${array[@]}" >> /etc/environment.new
mv /etc/environment.new /etc/environment

Here we use printf "%s\n" "${array[@]}" to print the array on a new line (cfr bash print array elements on separate lines) and we use the cut command to retrieve everything before =.

Just use a single awk:

When you think of combinations of grep, cut, awk, sed, tr, ... you generally can replace it with a single awk. We will use awk to read 2 files. The first is /etc/environment and the second is the file we can generate from the printf command. We will use an awk array in which we store the values per key where the key is the part before the =. So the =-sign will be our delimiter. Furthermore, if we encounter a variable with an identical key, it will be overwritten. This allows us to update the values with new values.

awk -F'=' '{a[$1]=$0}END{for(i in a) print a[i] }' /etc/environment <(printf "%s\n" "${array[@]}")

If you don't really use the array further in your script, you can just use a HERE document to achieve the same:

cat - <<EOF awk -F'=' '{a[$1]=$0}END{for(i in a) print a[i] }' /etc/environment -
http_proxy=http://username:password@proxy:8080
https_proxy=http://username:password@proxy:8080
ftp_proxy=ftp://username:password@proxy:8080
no_proxy="localhost, 127.0.0.1"
EOF

Awk output not consistent

Update

1 Answers1