How to make sed command handle the escape characters in the input parameters that are not user-typed?

Question

I've a file inside which I have a placeholder text for a password. I'm now trying to find and replace the placeholder text with the actual password. The text looks like below:

USER_GIVEN_PASSWORD=<my_password>

Let's say my password is: ABC&12345

I'm using the below command inside a script to replace this:

sed -i "s/<my_password>/$1/g" file.txt

I pass the input to my script as below:

sh password_replace.sh ABC&12345

My expected output is:

USER_GIVEN_PASSWORD=ABC&12345

But I'm getting the below output:

USER_GIVEN_PASSWORD=ABC<my_password>12345

Clearly, I'm doing something wrong with the & symbol present in my password. So, when I tried with escaping & in my input as follows, it actually works:

sh password_replace.sh ABC'\&'12345

But the problem is I should not adjust the input parameter to pass an escape character because the password won't be manually typed. It will automatically come from something like the Azure key vault as the input to my script.

So, I need to make the sed command itself handle the incoming special characters.

Can someone please help me achieve this?

Does this answer your question? [Is it possible to escape regex metacharacters reliably with sed](https://stackoverflow.com/questions/29613304/is-it-possible-to-escape-regex-metacharacters-reliably-with-sed) — Wiktor Stribiżew, Dec 04 '20 at 14:37
Also, `&` cannot be passed unquoted as a part of a parameter in the shell, as it is a control operator. — choroba, Dec 04 '20 at 14:39
@Wiktor Stribizew: I don't think I can implement regex in the parameter when passing it. Because, as mentioned in the question, the whole password comes from somewhere else and it might or might not contain `&` or any other special character. Aso, I'm pretty new to sed and regex and the question you linked has gone over my head :-( — LearneR, Dec 04 '20 at 14:54
If your shell program has the password in a variable, and the shell is Bash, then you can use parameter substitution to ready it for `sed`: `pass=${pass//\\/\\\\}; pass=${pass//&/\\&}`. — Toby Speight, Dec 04 '20 at 15:46

Raman Sailopal · Answer 1 · 2020-12-04T16:08:21.957

0

As others have commented, & is a reserved word in sed and will need to be "escaped" in order to be used in substitute text. An alternative approach would be to use awk (GNU awk in this case):

awk -v pass="$pass" -v confpass="$confpass" -v schepass1="$schepass1" -v schepass2="$schpass2" '/USER_GIVEN_PASS/ { match($0,"<my_password>");$0=substr($0,1,RSTART-1)pass""substr($0,RSTART+RLENGTH) } /CONFIRM_USER_GIVEN_PASSWORD/ { match($0,"<my_password>");$0=substr($0,1,RSTART-1)confpass""substr($0,RSTART+RLENGTH) } /SCHEMA1_PASSWORD/ { match($0,"<db_password>");$0=substr($0,1,RSTART-1)schpass1""substr($0,RSTART+RLENGTH) } /SCHEMA2_PASSWORD/ { match($0,"<db_password>");$0=substr($0,1,RSTART-1)schpass2""substr($0,RSTART+RLENGTH) }1' file

Pass the password to awk as a variable "pass". When the line containing "USER_GIVEN_PASS" is encountered, match the position of the "holder" using awk's match function. Set the line ($0) equal to the string up to the place holder, pass, then the rest of the string.

If you have later versions of GNU awk, you can use -i to commit the changes back to the file once you are happy.

Otherwise:

awk -v pass="$pass" '/USER_GIVEN_PASS/ { match($0,"<my_password>");$0=substr($0,1,RSTART-1)pass""substr($0,RSTART+RLENGTH) }1' file > tmpfile && mv -f tmpfile file

edited Dec 04 '20 at 16:08

answered Dec 04 '20 at 15:23

Raman Sailopal

12,320
2
11
18

This works well but only when I have one instance to replace. What if I have multiple such placeholders all in the same file? For example: `USER_GIVEN_PASSWORD=` `CONFIRM_USER_GIVEN_PASSWORD=` `SCHEMA1_PASSWORD=` `SCHEMA2_PASSWORD=` – LearneR Dec 04 '20 at 16:00
Same logic. Pass in more variables and then follow the same process. – Raman Sailopal Dec 04 '20 at 16:02
1

I've amended the answer to give you an idea. – Raman Sailopal Dec 04 '20 at 16:08
That's really just shifting the problem from `&` to ```\``` rather than solving it as you still end up with a character that needs to be escaped. – Ed Morton Dec 04 '20 at 20:54
Using `pass""` is no different from `pass ` (space at the end) in string concatenation btw. – Ed Morton Dec 04 '20 at 21:07

Ed Morton · Accepted Answer · 2020-12-04T21:06:09.910

Sed doesn't understand literal strings, just regexps and backreference-enabled replacement text. See Is it possible to escape regex metacharacters reliably with sed for the lengths you have to go to to try to get sed to behave as if you were using literal strings.

Just use the literal string functions in awk instead:

$ new='ABC&12\345' awk '
    BEGIN { old="<my_password>"; lgth=length(old); new=ENVIRON["new"] }
    s = index($0,old) { $0 = substr($0,1,s-1) new substr($0,s+lgth) }
1' file
USER_GIVEN_PASSWORD=ABC&12\345

That will work for any character, no need to do any escaping.

In your shell script where you're trying replace with $1 you'd just do:

new="$1" awk 'script' file

If you're using GNU awk you can use -i inplace just like you can use -i with GNU sed:

new="$1" awk -i inplace 'script' file

or with any awk:

tmp=$(mktemp)
new="$1" awk 'script' file > "$tmp" && mv "$tmp" file

It's important for new=... to be at the start of the same line as the awk command starts on so ENVIRON[] can access it without it having to be exported.

How to make sed command handle the escape characters in the input parameters that are not user-typed?

2 Answers2