How to use variables in a sed command line?

Question

I have this configuration file that has

# some other configuration settings
.....
wrapper.java.classpath.1=/opt/project/services/wrapper.jar
wrapper.java.classpath.2=/opt/project/RealTimeServer/RTEServer.jar
wrapper.java.classpath.3=/opt/project/mysql-connector-java-5.1.39-bin.jar
.....
# some other configuration settings

and I want it to look like this

# some other configuration settings
.....
wrapper.java.classpath.1=/opt/project/services/wrapper.jar
wrapper.java.classpath.2=/opt/project/RealTimeServer/RTEServer.jar
wrapper.java.classpath.3=/opt/project/mysql-connector-java-5.1.39-bin.jar
wrapper.java.classpath.4=/opt/project/RealTimeServer/some_other.jar
.....
# some other configuration settings

So I wrote this bash shell

#!/bin/bash

CONF_FILE=$1
JAR_FILE=$2
DIR=$3

# Get the last wrapper.java.classpath.N=/some_path line
CLASSPATH=`awk '/classpath/ {aline=$0} END{print aline}' $CONF_FILE`
echo $CLASSPATH

# Get the left side of the equation
IFS='=' read -ra LS <<< "$CLASSPATH"

# Get the value of N
NUM=${LS##*\.}

# Increment by 1
NUM=$((NUM+1))
echo $NUM

NEW_LINE="wrapper.java.classpath.$NUM=$DIR/$JAR_FILE"
echo $NEW_LINE

# Append classpath line to conf file
sed "/$CLASSPATH/a \\${NEW_LINE}" $CONF_FILE

I call it this way

./append_classpath.sh some_file.conf some_other.jar /opt/project/RealTimeServer

But I get

sed: -e expression #1, char 28: unknown command: `o'

Give us a all the info. The test file, the variables, and what OS and shell you are using. — Bill Doughty, Aug 19 '16 at 17:04
Just a shot in the dark: Try '${some_string}'. The $ appears to only see the first letter of your variable name. Same for ${some_line}. — steffen, Aug 19 '16 at 17:09
Karoly is correct :-). @ChrisF - always quote your shell variables - `"$some_file"`, not `$some_file`! If you don't know why then google it stat! Also do not use all-upper-case variable names for non-exported variables to avoid clashes with existing variables. — Ed Morton, Aug 19 '16 at 17:25
You need proper quoting, try `sed '/'"$CLASSPATH"'/a\'"${NEW_LINE}" "$CONF_FILE"` — David C. Rankin, Aug 20 '16 at 19:34

score 0 · Answer 1 · edited May 23 '17 at 12:15

I just saw your shell script. A shell is an environment from which to call tools, it is NOT a tool to manipulate text. The standard, general purpose UNIX tool to manipulate text is awk. Your entire shell script can be reduced to:

$ dir="/opt/project/RealTimeServer"
$ jar_file="some_other.jar"
$ awk -v new="$dir/$jar_file" 'p~/classpath/ && !/classpath/{match(p,/([^=]+\.)([0-9]+)=/,a); print a[1] (++a[2]) "=" new} {print; p=$0}' file
# some other configuration settings
.....
wrapper.java.classpath.1=/opt/project/services/wrapper.jar
wrapper.java.classpath.2=/opt/project/RealTimeServer/RTEServer.jar
wrapper.java.classpath.3=/opt/project/mysql-connector-java-5.1.39-bin.jar
wrapper.java.classpath.4=/opt/project/RealTimeServer/some_other.jar
.....
# some other configuration settings

The above uses GNU awk for the 3rd arg to match(). Read the book Effective Awk Programming, 4th Edition, by Arnold Robbins if you will ever have to manipulate text in a UNIX environment.

Now back to your question:

This is the syntax for what you are TRYING to do:

sed '/'"$some_string"'/a '"$some_line" "$some_file"

BUT DON'T DO IT or you'll be condemning yourself to cryptic, non-portable, unmaintainable, peeling the onion, escaping-everything hell (see Is it possible to escape regex metacharacters reliably with sed)!

sed is for simple subsitutions on individual lines, that is all. For anything else, e.g. what you are attempting, you should be using awk:

awk -v regexp="$some_string" -v line="$some_line" '{print} $0~regexp{print line}' file

Note that although your shell variable is named "some_string" you were using it in a regexp context (all you can do with sed) so I used it in a regexp context in the awk command too and named the awk variable "regexp" rather than "string" for clarity (it's just a variable name, though, no hidden meaning).

If you really DID want it treated as a string rather than a regexp then that'd be:

awk -v string="$some_string" -v line="$some_line" '{print} index($0,string){print line}' file

The only caveat to the above is that backslashes in the shell variables will be expanded when the awk variables are initialized from them so \t, for example, would become a literal tab character. If that's undesirable let us know and we can provide an alternative syntax for initing the awk variables that does not expand backslashes, see http://cfajohnson.com/shell/cus-faq-2.html#Q24.

Thanks, but I tried this and it did NOT append the line. Note that I have lines before and after the wrapper lines. I'll edit my post to reflect this. — Chris F, Aug 19 '16 at 17:51
Also you missed the matching single quote here correct? sed '/'"$some_string"'/a '"$some_line"' "$some_file". I did use that too to no avail, event after I fixed the syntax. — Chris F, Aug 19 '16 at 17:57
Right, if your real input doesn't look like your sample input then all bets are off. I might've missed a quote on the sed command, doesn't matter as it shouldn't be used anyway. Actually it's just one quote too many. — Ed Morton, Aug 19 '16 at 18:04
I tweaked the awk command to work with your new input. If it doesn't work for you, make sure you are using GNU awk 4.0 or newer. If not, get it. If that's absolutely impossible then use a combination of sub() and substr() instead of the array a[]. To be clear - I showed the sed syntax for what you were trying to do, I didn't mean to imply it would actually work (see what I said about peeling onions, etc.). — Ed Morton, Aug 19 '16 at 18:10
Thanks, but unfortunately the new awk command didn't append the line I want. I have gawk v4.0.2 on CentOS 7.2 — Chris F, Aug 19 '16 at 19:34
Then once again your posted sample input must not match your real input since you can see in my answer the script adding the line you want to your sample input. The script does work for what you've shown us so far. [edit] your question to show minimal sample input that the command is failing for, the command you are running (including the setting of the 2 shell variables, and the output it produces. — Ed Morton, Aug 20 '16 at 15:53

score 0 · Answer 2 · answered Aug 19 '16 at 19:05

The sed command will have problems with the slashes in your variables. Look for some unique delimiter such as a # and try something like

CLASSPATH="wrapper.java.classpath.4=/opt/project/RealTimeServer/some_other.jar"
NEW_LINE="wrapper.java.classpath.5=your/data.rar"

echo "# some other configuration settings
.....
wrapper.java.classpath.1=/opt/project/services/wrapper.jar
wrapper.java.classpath.2=/opt/project/RealTimeServer/RTEServer.jar
wrapper.java.classpath.3=/opt/project/mysql-connector-java-5.1.39-bin.jar
wrapper.java.classpath.4=/opt/project/RealTimeServer/some_other.jar
.....
# some other configuration settings

Some more config lines
"  | sed "s#${CLASSPATH}#&\n${NEW_LINE}#"

It will have problems with more characters than just slashes, e.g. any BRE regexp metacharacter, backslash followed by a number, or `&`. See http://stackoverflow.com/q/29613304/1745001 for what it takes to do this robustly with sed. — Ed Morton, Aug 20 '16 at 15:51

score 0 · Answer 3 · answered Aug 20 '16 at 17:38

This is a one-pass pure bash solution - it should be fine if the configuration file is not huge

pfx=wrapper.java.classpath.
while IFS= read -r line; do
  if [[ $line == $pfx*=* ]]; then
    lastclasspath=$line
  elif [[ -n $lastclasspath ]]; then
    newline=${lastclasspath#$pfx}
    num=${newline%%=*}
    newline="$pfx$((num+1))=$DIR/$JAR_FILE"
    echo "$newline"
    unset lastclasspath
  fi
  echo "$line"
done <$CONF_FILE

How to use variables in a sed command line?

3 Answers3