How to escape history expansion exclamation mark ! inside a double quoted string?

Question

EDIT: the command substitution is not necessary for the surprising behavior, although it is the most common use case. The same question applies to just echo "'!b'"

b=a

# Enable history substitution.
# This option is on by default on interactive shells.
set -H

echo '!b'
# Output: '!b'
# OK. Escaped by single quotes.

echo $(echo '!b')
# Output: '!b'
# OK. Escaped by single quotes.

echo "$(echo '$b')"
# Output: '$b'
# OK. Escaped by single quotes.

echo "$(echo '!b')"
# Output: history expands
# BAD!! WHY??

In the last example, what is the best way to escape the !?
Why was it not escaped even if I used single quotes, while echo "$(echo '$b')" was? What is the difference between ! and $?
Why was does echo $(echo '!b') (no quotes) work? (pointed by @MBlanc).

I would prefer to do this without:

set +H as I would need set -H afterwards to maintain shell state

backslash escapes, because I need one for every ! and it has to be outside the quotes:

echo "$(echo '\!a')"
# '\!a'.
# No good.

echo "$(echo 'a '\!' b '\!' c')"
# a ! b ! c
# Good, but verbose.

echo $(echo '!b') (no quotes), because the command could return spaces.

Version:

bash --version | head -n1
# GNU bash, version 4.2.25(1)-release (i686-pc-linux-gnu)

It also works for me. I would prefer a quoting solution so I don't have to escape every `!` in the string, but it works. Please add your comment to an answer. There is still the why question. — Ciro Santilli OurBigBook.com, Mar 02 '14 at 09:04
Removing the outermost quotes also prevents substitution. I'd go with `set +H` myself if I were writing a script. I hope a bash expert sees this and enlightens us. Good luck! :) — MBlanc, Mar 02 '14 at 09:17
@KarolyHorvath I also thought of this reasoning, but why does `echo "$(echo -e '! \bb')"` or `echo "$(echo -e "! \bb")"` work? Don't both of these lead to `echo "!b"`? — devnull, Mar 02 '14 at 09:57
@devnull those work because the space after `! ` prevents `!` expansion. Thats clear in the man pages. — Ciro Santilli OurBigBook.com, Mar 02 '14 at 10:01
@cirosantilli I was referring to the _result_ of command substitution. — devnull, Mar 02 '14 at 10:02
"Several characters inhibit history expansion if found immediately following the history expansion character, even if it is unquoted: space, tab, newline, carriage return, and =." — Karoly Horvath, Mar 02 '14 at 10:02
@KarolyHorvath but why `echo "$(echo '$b')"` gives `$b` and not `a`? (it expands "only once", while the `!` command expands twice) — Ciro Santilli OurBigBook.com, Mar 02 '14 at 10:03
@devnull Can you provide an example without `-e`? With `-e`, `\b` is backspace, and that makes by brain twist even more. Without the `-e` the result is different. — Ciro Santilli OurBigBook.com, Mar 02 '14 at 10:14
@cirosantilli `-e` enables interpretation of backslash escapes; that would explain different results with and without it. — devnull, Mar 02 '14 at 10:35
@cirosantilli, if this is a script, history expansion is off by default, so you don't need to worry about this at all. If this is **not** a script, the issue belongs on SuperUser, not StackOverflow, so it's off-topic. — Charles Duffy, Mar 02 '14 at 17:22
@cirosantilli ...by the way, I strongly suggest turning off history expansion for your interactive shells: `set -H` in your `bashrc`. It's a feature that hails from a day when the readline library didn't exist yet and command-line editing was far more cumbersome. — Charles Duffy, Mar 02 '14 at 17:24
@CharlesDuffy: Good advice, but you meant `set +H` to turn history expansion _off_ - `set -H` turns it _on_. — mklement0, Nov 13 '15 at 03:24
Related: [Difference between single and double quotes in Bash](https://stackoverflow.com/q/6697753/6862601). — codeforester, May 13 '19 at 19:54
See also https://stackoverflow.com/questions/11816122/echo-fails-event-not-found — tripleee, Dec 20 '22 at 08:58
Also see [Bash Pitfalls #23 (echo "Hello World!")](https://mywiki.wooledge.org/BashPitfalls#echo_.22Hello_World.21.22). — pjh, Mar 22 '23 at 15:06

score 23 · Accepted Answer · answered Mar 02 '14 at 17:19

23

In your last example,

echo "$(echo '!b')"

the exclamation point is not single-quoted. Because history expansion occurs so early in the parsing process, the single quotes are just part of the double-quoted string; the parser hasn't recognized the command substitution yet to establish a new context where the single quotes would be quoting operators.

To fix, you'll have to temporarily turn off history expansion:

set +H
echo "$(echo '!b')"
set -H

answered Mar 02 '14 at 17:19

chepner

497,756
71
530
681

Thanks for you answer! What are the names of the relevant parsing stages? Where can I get that info on the man page? – Ciro Santilli OurBigBook.com Mar 02 '14 at 17:49
This does even happen when a password is read from command line. Thank you! – Michael-O Nov 12 '14 at 10:48
2

@CiroSantilli Unfortunately, the exact order of operations is scattered throughout the man page. Under HISTORY EXPANSION, it is mentioned that history expansion occurs immediately after the line is read, but before any word-splitting occurs. This implies that `!b` would be replaced by the appropriate command prior to any other structures in the line being recognized. – chepner Nov 12 '14 at 13:19
Great answer; strictly speaking, `!` parsing doesn't even respect double-quotes per se, and considers any run of non-whitespace characters following `!` the history-expansion argument, including `"`; for instance, `echo foo!"` - despite imbalanced double-quotes - is considered a valid command, as is `echo foo!bar"baz`. – mklement0 Nov 13 '15 at 04:37
The quoting of `!` raises some troubling issues for software that generates text for interpretation by bash. With the exception of that one character, the quoting rules are simple. Protecting against exclamation point injection into a shell with history and history expansion enabled is going to be … an interesting challenge. – Eric Aug 28 '16 at 01:05
This is a bug in bash 4.1, fixed in 4.2 (released five years ago, 2011). – jthill Dec 21 '16 at 17:44
@jthill What bug are you referring to? `echo "$(echo '!b')"` still subjects `!b` to history expansion in 4.3. (It does appear changed in 4.4, though.) – chepner Dec 21 '16 at 17:47
@chepner Hunh. I got here from http://stackoverflow.com/a/17155384/1290731 and with the command there, `sed "$(sed -n /a/= file | sed '$!d;s/$/ s,a,c,/' )" file `, history expansion _doesn't_ take place since 4.2 (I just finished running a bisect to find it). I didn't think to check this one. It still does, as you say. – jthill Dec 21 '16 at 17:54
Might have been fixed in 4.2, reintroduced in 4.3, and fixed again in 4.4 :) – chepner Dec 21 '16 at 17:54
actually, that's possible, because my "it still does" was done with the `git bisect old` shell. on 4.4.5 I get `!b`. Hang on, I'll find where this was fixed. I wish they'd just used git so I could just git log --grep for it. – jthill Dec 21 '16 at 17:57
`git://git.sv.gnu.org/bash.git` – chepner Dec 21 '16 at 17:58
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/131183/discussion-between-jthill-and-chepner). – jthill Dec 21 '16 at 18:00
This is a bug, fixed in bash 4.4 – jthill Dec 21 '16 at 18:17
@jthill That's probably worth posting as a new answer, especially if you can provide an explanation for *why* it was considered a bug and what change was made to protect the history expansion. – chepner Dec 21 '16 at 18:25
Is not fixed on `GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)` version! – Idemax Nov 01 '18 at 15:01
1

@MarceloFilho No, it is not, as discussed in the previous comments and leading to [jthill's answer](https://stackoverflow.com/a/41273889/1126841). – chepner Nov 01 '18 at 15:03

score 4 · Answer 2 · answered Dec 22 '16 at 00:04

This was repeatedly reported as a bug, most recently against bash 4.3 in 2014, for behavior going back to bash 3.

There was some discussion whether this constituted a bug or expected but perhaps undesirable behavior; it seems the consensus has been that, however you want to characterize the behavior, it shouldn't be allowed to continue.

It's fixed in bash 4.4, echo "$(echo '!b')" doesn't expand, echo "'!b'" does, which I regard as proper behavior because the single quotes are shell syntax markers in the first example and not in the second.

John B · Answer 3 · 2018-02-14T06:43:58.620

2

If History Expansion is enabled, you can only echo the ! character if it is put in single quotes, escaped or if followed by a whitespace character, carriage return, or =.

From man bash:

   Only backslash (\) and single quotes can  quote  the  history
   expansion character.

   Several  characters inhibit history expansion if found immediately fol-
   lowing the history expansion character, even if it is unquoted:  space,
   tab,  newline,  carriage return, and =.

I believe the key word here is “Only”. The examples provided in the question only consider the outer most quotes being double quotes.

edited Feb 14 '18 at 06:43

answered Mar 02 '14 at 11:17

John B

3,566
1
16
20

Thank you John B, but as implied in the question I knew about the single quotes, and as stated in the comments I also knew about the characters that come after it. The main question is: why does the inner single quote does not work? – Ciro Santilli OurBigBook.com Mar 02 '14 at 11:18
but then why `echo "$(echo '$b')"` does not expand (gives `$b`) while `echo "$(echo '!b')"` does (does history expansion)? – Ciro Santilli OurBigBook.com Mar 02 '14 at 11:29
@cirosantilli I think that happens because Bash treats `!` differently than word splitting when it comes to quoting. As the man page specificity of quoting implies, it is much more strict with the `!` character. – John B Mar 02 '14 at 11:57
Do you mean: "If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed"? That is the only specific thing about quoting and `!`. But it twists my brain. On the above examples the backslash was removed at some point. To actually understand this I would need a breakdown of the relevant manpage line + step by step of the expansions done. – Ciro Santilli OurBigBook.com Mar 02 '14 at 12:05

score 2 · Answer 4 · answered Dec 18 '19 at 15:58

Sometimes you need to make a small addition to a big command pipe

The OP's "Good, but verbose" example is actually pretty awesome for many cases.

Please forgive the contrived example. The whole reason I need such a solution is that I have a lot of distracting, nested code. But, it boils down to: I must do a !d in sed within a double quoted bash command expansion.

This works

$ ifconfig | sed '/inet/!d'
inet 127.0.0.1 netmask 0xff000000
…

This does not

$ echo "$(ifconfig | sed '/inet/!d')"
-bash: !d': event not found

This is a simplest compromise

$ echo "$(ifconfig | sed '/inet/'\!'d')"
inet 127.0.0.1 netmask 0xff000000
…

Using the compromise allows me to insert a few characters into the existing code and produce a Pull Request that anyone can understand… even though resulting code is more difficult to understand. If I did a complete refactor, the code reviewers would have a much more challenging time verifying it. And of course this bash has no unit tests.

Boris Verkhovskiy · Answer 5 · 2023-08-24T20:19:35.277

1

Close the double quote, put the ! in single quotes ' immediately after (without a space before or after it) and then open the double quote again:

$ echo "this is "'!'"how you do it"
this is !how you do it

edited Aug 24 '23 at 20:19

answered Mar 22 '23 at 09:00

Boris Verkhovskiy

14,854
11
100
103

How to escape history expansion exclamation mark ! inside a double quoted string?

5 Answers5

Sometimes you need to make a small addition to a big command pipe

This works

This does not

This is a simplest compromise

Linked

Related