1

I am fighting with sed to do a substitution where the substitute string contains slashes. This general topic has been discussed on stack overflow before. But, AFAICT, I have anew wrinkle that hasn't been addressed in previous questions.

Let's say I have a file, ENVIRO.tpml, which has several lines, one of which is

Loaded modules: SUPPLY_MODULES_HERE

I want to replace SUPPLY_MODULES_HERE in an automated fashion with a list of loaded modules. (At this point, if anyone has a better way to do this than sed, please let me know!) My first effort here is to define an environment variable and use sed to put it into the file:

> modules=$(module list 2>&1)
> sed "s/SUPPLY_MODULES_HERE/${modules}/" ENVIRO.tmpl > ENVIRO.txt

(The 2>&1 being needed because module list sends its output to STDERR, for reasons I can't begin to understand.) However, as is often the case, the modules have slashes in them. For example

> echo ${modules}
gcc/9.2.0  mpt/2.20

The slashes kill my command because sed can't understand the expression and thinks my substitution command is "unterminated".

So I do the usual thing and use some other character for the command delimiter:

> modules=$(module list 2>&1)
> sed "s|SUPPLY_MODULES_HERE|${modules}|" ENVIRO.tmpl > ENVIRO.txt

and I still get an "unterminated 's'" error.

So I replace double quotes with single quotes:

> sed 's|SUPPLY_MODULES_HERE|${modules}|' ENVIRO.tmpl > ENVIRO.txt

and now I get no error, but the line in ENVIRO.txt looks like

Loaded modules: ${modules}

Not what I was hoping for.

So, AFAICT, I need double quotes to expand the variable, but I need single quotes to make the alternative delimiters work. But I need both at the same time. How do I get this?

UPDATE: Gordon Davisson's comment below got to the root of the matter: "echo ${modules} can be highly misleading". Examining $modules with declare -p shows that it actually has a newline (or, more generally, some kind of line break) in it. What I did was add an extra step to extract newlines out of the variable. With that change, everything worked fine. An alternative would be to convince sed to expand the variable with line breaks and substitute it as such into the text, but I haven't been able to make that work. Any takers?

bob.sacamento
  • 6,283
  • 10
  • 56
  • 115
  • I don't think you "need single quotes to make the alternative delimiters work". Your example seemed to work for me. Are you testing it with that actual example? `modules="gcc/9.2.0 mpt/2.20";echo Loaded modules: SUPPLY_MODULES_HERE | sed "s|SUPPLY_MODULES_HERE|${modules}|"` Or do you have a different scenario where the modules have a pipe in them? – EdmCoff Nov 03 '21 at 17:10
  • 2
    What's the actual contents of `$modules`? [`echo ${modules}` can be highly misleading](https://stackoverflow.com/questions/29378566/i-just-assigned-a-variable-but-echo-variable-shows-something-else), so try `declare -p modules | LC_ALL=C cat -v` instead. – Gordon Davisson Nov 03 '21 at 17:24
  • There’s nothing obviously wrong with the version using double quotes and pipes. Prepend `echo`, removing redirection, and post that output, so we can see what sed is trying to execute. – dan Nov 03 '21 at 17:26
  • If your templates have several substitutions to perform or more complex operations then you could directly use a templating language like PHP or ERB (Ruby). – Fravadona Nov 03 '21 at 21:19
  • @GordonDavisson You might be interested in my update. – bob.sacamento Nov 03 '21 at 22:22
  • What is the output of `modules="$(module list 2>&1)"; declare -p modules` ? – Fravadona Nov 03 '21 at 23:40
  • Does https://stackoverflow.com/questions/407523/escape-a-string-for-a-sed-replace-pattern answer your question? – KamilCuk Nov 03 '21 at 23:41

1 Answers1

4

sed is not the best tool here due to use of regex and delimiters.

Better to use awk command that doesn't require any regular expression.

awk -v kw='SUPPLY_MODULES_HERE' -v repl="$(module list 2>&1)" '
n = index($0, kw) {
   $0 = substr($0, 1, n-1) repl substr($0, n+length(kw))
} 1
' file
  • index function uses plain string search in awk.
  • substr function is used to get substring before and after the search keyword.
anubhava
  • 761,203
  • 64
  • 569
  • 643