Multiline syntax for piping a heredoc; is this portable?

Question

I'm familiar with this syntax:

cmd1 << EOF | cmd2
text
EOF

but just discovered that bash allows me to write:

cmd1 << EOF |
text
EOF
cmd2

(the heredoc is used as input to cmd1, and the output of cmd1 is piped to cmd2). This seems like a very odd syntax. Is it portable?

I came here to find a good way of splitting this into multiple lines: `big-long-command1 with lots of args << EOF | big-long-command2 with lots of args`. The "odd syntax" seems like the best way. — PaulC, Dec 30 '14 at 21:02
One convenient use case for this is when you're trying to convert a table that is space delimited into one that is tab-delimited so you can paste it in Google Spreadsheets. You won't have to create a temporary file. — Sridhar Sarnobat, Jan 05 '18 at 00:08
The 1st one didn't work for me in z-shell. I don't like the 2nd one because it alienates the | from the command, losing the idiomacy (?) of shell pipelines. — Sridhar Sarnobat, Jan 05 '18 at 01:12

Ned Deily · Accepted Answer · 2012-05-20T21:04:47.960

Yes, the POSIX standard allows this. According to the 2008 version:

The here-document shall be treated as a single word that begins after the next <newline> and continues until there is a line containing only the delimiter and a <newline>, with no <blank> characters in between. Then the next here-document starts, if there is one.

And includes this example of multiple "here-documents" in the same line:

cat <<eof1; cat <<eof2
Hi,
eof1
Helene.
eof2

So there is no problem doing redirections or pipes. Your example is similar to something like this:

cat file |
cmd

And the shell grammar (further down on the linked page) includes these definitions:

pipe_sequence    :                             command
                 | pipe_sequence '|' linebreak command

newline_list     :              NEWLINE
                 | newline_list NEWLINE
                 ;
linebreak        : newline_list
                 | /* empty */

So a pipe symbol can be followed by an end-of-line and still be considered part of a pipeline.

Jens · Answer 2 · 2014-04-24T21:06:20.510

28

Yes it's in the POSIX shell grammar. You can also have more than one here-doc for the same command (some other examples use two cat invocations, but this works as well):

cat <<EOF1 <<EOF2
first here-doc
EOF1
second here-doc
EOF2

This is contrived (using 2 here-docs for stdin), but if you think of providing input for different file descriptors it immediately makes sense.

There's also the possibility to drop the cat entirely. Why not make the here-document directly available to cmd:

cmd << EOF
input
here
EOF

edited Apr 24 '14 at 21:06

answered May 20 '12 at 21:10

Jens

69,818
15
125
179

1

``` cat < – user1424739 Jun 01 '15 at 00:29
@user1424739 It does work in current zsh and bash. The ash and ksh93 seem to output only the second here doc. – Jens Jun 01 '15 at 07:30
Why the downvote? If there's something inaccurate, please give me opportunity to remedy. – Jens Jan 01 '17 at 21:15
This is pretty sweet when using `sudo tee /etc/securefile.conf < – dragon788 Jul 19 '17 at 23:18
3

On what bash version does it work ? Using bash 4.4.19 (on ubuntu 18.04.02) and bash 5.0 (docker image), I only got the second here-doc. Or maybe there's a specific option ? – huelbois Jul 05 '19 at 09:13
@huelbois Same here with bash 5.1.8 – Cigarette Smoking Man Dec 04 '21 at 09:37

Tomas · Answer 3 · 2018-01-11T13:46:51.837

19

Hmm, I suppose yes, according to the test in bash in POSIX mode:

$ bash --posix
$ cat <<EOF |
> ahoj
> nazdar
> EOF
> sed 's/a/b/'
bhoj
nbzdar

edited Jan 11 '18 at 13:46

answered Aug 12 '11 at 21:17

Tomas

57,621
49
238
373

Just one other tiny note: do not put any spaces after the closing `EOF`. The prompt will behave strangely and you'll wonder what the hell is wrong – Sridhar Sarnobat Jan 05 '18 at 00:03
2

Running bash in POSIX-mode shuts off *some* extensions, but not by any means even nearly all of them. As such, while this answer is correct in terms of what POSIX allows, its reasoning doesn't support that very effectively. – Charles Duffy Jan 05 '18 at 00:09

score 9 · Answer 4 · answered Mar 21 '19 at 12:45

9

Hi, check this, for example

#!/bin/sh
( base32 -d | base64 -d )<<ENDOFTEXT
KNDWW42DNNSHS5ZXPJCG4MSVM5MVQVT2JFCTK3DELBFDCY2IIJYGE2JUJNHWS22LINVHQMCMNVFD
CWJQIIZVUV2JOVNEOVJLINTW6PIK
ENDOFTEXT

regards

answered Mar 21 '19 at 12:45

buc

91
1
1

Multiline syntax for piping a heredoc; is this portable?

4 Answers4

Linked