639

I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.

e.g. repo forall -c '....$variable'

In this format, $ is escaped and the variable is not expanded.

I tried the following variations but they were rejected:

repo forall -c '...."$variable" '

repo forall -c " '....$variable' "

" repo forall -c '....$variable' "

repo forall -c "'" ....$variable "'"

If I substitute the value in place of the variable the command is executed just fine.

Please tell me where am I going wrong.

codeforester
  • 39,467
  • 16
  • 112
  • 140
Rachit
  • 6,531
  • 3
  • 16
  • 9
  • 55
    `repo forall -c ' ...before... '"$variable"' ...after...'` – n. m. could be an AI Dec 10 '12 at 11:25
  • 3
    @n.m. the single quotes are a part of the repo command. i dont think this will work – Rachit Dec 10 '12 at 11:27
  • 1
    `bash` eats single quotes. Either you are not in `bash`, or single quotes are not a part of the `repo` command. – n. m. could be an AI Dec 10 '12 at 11:31
  • Not sure I get you, but in case single quotes need to be there, wouldn't repo forall -c "' ...before... "$variable" ...after...'" work? – ecv Dec 10 '12 at 11:48
  • Its a bash script running in bash shell and repo forall command takes parameters inside single quotes. If I substitute the actual value, the command runs fine. – Rachit Dec 10 '12 at 11:49
  • @Kevin. No it is giving the error: 'before value after': 'before value after' No such file or directory. – Rachit Dec 10 '12 at 11:53
  • I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones. If you could provide me with the real case, I think I could be of greater help. – ecv Dec 10 '12 at 12:59
  • BTW, you may be running it from a bash shell/script, but repo commands are run in an sh shell, as per documentation: "-c: command and arguments to execute. The command is evaluated through /bin/sh and any arguments after it are passed through as shell positional parameters." <- [link](http://source.android.com/source/using-repo.html#forall) – ecv Dec 10 '12 at 13:04
  • Another approach that could work would be passing the needed variable to repo execution, which seems to successfully spread it onto the spawned sh's. Like so: variable=$variable ./repo forall -c 'blahblah $variable blahblah'. The single quotes enclosing the full command are dealt with by repo, and $variable is actually evaluated. – ecv Dec 10 '12 at 13:15
  • @Kevin: Yes you were right. repo forall takes double quotes as well instead of single quotes. Stupid of me to not check that once. Thanks for your help. If you will post your answer I will accept it as correct one. – Rachit Dec 11 '12 at 06:06
  • @Kevin: Also I read up on the execution documentation and I think you are right. but no time to check up on that. I will go with double quotes. Thank you very much!!! – Rachit Dec 11 '12 at 06:07
  • See: [Difference between single and double quotes in Bash](https://stackoverflow.com/questions/6697753/difference-between-single-and-double-quotes-in-bash). – codeforester Dec 21 '17 at 18:43
  • And also more broadly [When to wrap quotes around a shell variable?](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) – tripleee Oct 04 '18 at 11:13

8 Answers8

982

Inside single quotes everything is preserved literally, without exception.

That means you have to close the quotes, insert something, and then re-enter again.

'before'"$variable"'after'
'before'"'"'after'
'before'\''after'

Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?

Do not concatenate strings interpreted by a shell

You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).

Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.

For example, the following is very unsafe. DON'T DO THIS

script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"

If the contents of $myvar is untrusted, here is an exploit:

myvar='foo"; echo "you were hacked'

Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:

script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"

Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.

Jo So
  • 25,005
  • 6
  • 42
  • 59
  • @Jo.SO thatwill not work. because the repo command will take only the parameters till the first quotes are closed. Anything after that will be ignored for repo and generate another error. – Rachit Dec 11 '12 at 06:03
  • 9
    @Rachit - the shell doesn't work like that. `"some"thing' like'this` is all one word to the shell. Quotation marks don't terminate anything as far as parsing is concerned, and there's no way for a command called *by* the shell to tell what was quoted how. – Mark Reed Dec 11 '12 at 11:39
  • 3
    That second sentence ("you can't even escape single quotes") makes single quotes the black holes of the Shell. –  Jan 11 '16 at 23:40
  • @Evert: Don't know how to say it better. I've removed the sentence. – Jo So Mar 09 '17 at 15:13
  • @JoSo That's a shame: no the joke falls flat. –  Mar 09 '17 at 23:31
  • Why do you wrap the variable in "" in the first example? – Daniel Kaplan Jun 12 '21 at 22:43
  • 3
    @DanielKaplan The simple rule is *always quote your variables* or you'll conjure up nasal demons, like splitting on whitespace or even glob expansion. Always quote your variables. – Jo So Jun 16 '21 at 19:37
  • [This](https://stackoverflow.com/a/39463371/2172566) related answer was *tremendously* useful to me – forresthopkinsa Jul 10 '21 at 02:11
136

The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.

repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'

Explanation follows, if you're interested.

When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.

But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, strings in the shell are often called "words". But a shell word may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.

You can use a backslash in front of any character (including space or another backslash) to tell the shell to treat that character literally. But while you can do something like this:

reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier

...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.

Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:

reply="\"That'll be \$4.96, please,\" said the cashier"

Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.

Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:

reply='"That'\''ll be $4.96, please," said the cashier'

So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.

Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:

reply=$'"That\'ll be $4.96, please," said the cashier'

The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.

Mark Reed
  • 91,912
  • 16
  • 138
  • 175
  • 1
    Is the `$'string'` format POSIX compliant? Also, is there a name for it? – Wildcard Jan 21 '16 at 20:44
  • 1
    Link back - [What characters are required to be escaped in command line arguments?](http://unix.stackexchange.com/q/270977/135943) on the Unix & Linux stack exchange. – Wildcard Mar 20 '16 at 08:21
  • 1
    The most important info is: > quotation marks in the shell are not word delimiters; – Elan Hasson Feb 25 '22 at 06:25
  • @Wildcard $'string' is a non-POSIX extension, which I've seen referred to as "ANSI strings"; I've incorporated those facts into the answer. Such strings work in most modern Bourne-compatible shells: bash, ksh, and zsh all support them. But they don't work in strictly POSIX shells, or even mostly-POSIX shells like ash and dash. – Mark Reed Sep 20 '22 at 19:54
4

Below is what worked for me -

QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
melpomene
  • 84,125
  • 8
  • 85
  • 148
Manmohan
  • 178
  • 1
  • 6
  • 2
    I hope TBL_HDFS_DIR_PATH is not user provided input btw, this would allow a nice sql injection... – domenukk Nov 03 '17 at 17:37
  • 2
    Putting `QUOTE` in a separate variable is completely superfluous; single quotes inside double quotes are just a regular character. (Also, don't use upper case for your private variables.) – tripleee Mar 23 '19 at 13:08
2

EDIT: (As per the comments in question:)

I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.

ecv
  • 708
  • 7
  • 14
2

just use printf

instead of

repo forall -c '....$variable'

use printf to replace the variable token with the expanded variable.

For example:

template='.... %s'

repo forall -c $(printf "${template}" "${variable}")
AndrewD
  • 4,924
  • 3
  • 30
  • 32
  • 1
    This s broken; `printf` doesn't really buy you anything here, and failing to quote the command subsititution causes new quoting problems. – tripleee Mar 23 '19 at 13:09
0

Variables can contain single quotes.

myvar=\'....$variable\'

repo forall -c $myvar
Robert Columbia
  • 6,313
  • 15
  • 32
  • 40
user3734617
  • 99
  • 1
  • 1
  • 1
    They can, but this is not the right way to do it - you should still put `"$myvar"` in double quotes. – tripleee Mar 23 '19 at 13:10
-1

I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.

check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user@host "$check_var")
echo $getckvar

What do you get? A load of nothing.

Fix: escape \$3 in your print function.

Mr.G
  • 9
  • 3
  • Who are what is ibug and why did it put an extra \ in front of \$3? I'm not trying to escape the escape. I do appreciate the single quotes around the item in the page source. – Mr.G Dec 09 '22 at 22:37
-4

Does this work for you?

eval repo forall -c '....$variable'
drgnfr
  • 91
  • 1
  • 4
  • 10
    Does it for you? This question is five years old and already has some answers, even an accepted one... – Nico Haase Jan 23 '18 at 11:35
  • `eval` smushes all its arguments together into a single string before running that string through the parser; in this context, it causes far more problems than it solves. – Charles Duffy Dec 02 '22 at 16:56