Remove duplicate variables except last occurrence in bash script file

Question

I have the config file on local, whom I am appending some variables from different remote machines. The file content is as:

#!/bin/bash

name=bob
department=(Production)

name=alice
department=(R&D)

name=maggie
department=(Production R&D)

The latest values updated in the file are the last one. So the expected output in the config file should be:

#!/bin/bash
name=maggie
department=(Production R&D)

I want to remove the first two data of name and address except for the latest one which is last. But this should happen only if there are multiple same variables.

I referred and try this for my solution but not getting expected output: https://backreference.org/2011/11/17/remove-duplicates-but-keeping-only-the-last-occurrence/

Does this answer your question? [How do I delete an exported environment variable?](https://stackoverflow.com/questions/6877727/how-do-i-delete-an-exported-environment-variable) — Rob Evans, Nov 30 '20 at 09:54
@KamilCuk file is getting appended from the remote hosts where I cannot access the file to overwrite. Appending is done by redirecting the `sshpass` output the file. — Varsh, Nov 30 '20 at 10:05
It is not clear what you are looking for. Can you give a sample of what you see before the update and what you expect after? — Raman Sailopal, Nov 30 '20 at 10:15
`department=(Production R&D)` is a syntax error. You need some quotes. — William Pursell, Nov 30 '20 at 10:23
`is getting appended from the remote hosts where I cannot access the file to overwrite` If you can append (you have write access to the file), you can overwrite (in 99.99% of the cases). `Appending is done by redirecting the sshpass output the file` Great! So instead of `>>` use `>`. — KamilCuk, Nov 30 '20 at 10:23
@KamilCuk please refer my question why I append the file. https://stackoverflow.com/questions/64965726/access-and-append-local-file-from-remote-host-in-bash-script?noredirect=1#comment114856088_64965726 If `>` is used it overwrites everything. I also have some other credential variables which are not supposed to overwrite. — Varsh, Nov 30 '20 at 10:30

score 2 · Accepted Answer · answered Nov 30 '20 at 11:16

Would you please try the following:

tac file | awk '{                               # print "file" reversing the line order: last line first
    line = $0                                   # backup the line
    sub(/#.*/, "")                              # remove comments (not sure if comment line exists)
    if (match($0, /([[:alnum:]_]+)=/)) {        # look like an assignment to a variable
        varname = substr($0, RSTART, RLENGTH - 1)
                                                # extract the variable name (-1 to remove "=")
        if (! seen[varname]++) print line       # print the line if the variable is seen irst time
    } else {                                    # non-assignment line
        print line
    }
}' | tac                                        # reverse the lines again

Output:

#!/bin/bash



name=maggie
department=(Production R&D)

Please note the parser to extract variable names is a lousy one. You may need to tweak the code depending on the actual file.

Thank you for the feedback. You cannot overwrite the file with a redirect. Please redirect to another filename, say `file2` then rename it with `mv -f file2 file`. Good luck! — tshiono, Dec 01 '20 at 03:35

Remove duplicate variables except last occurrence in bash script file

1 Answers1