83

I'm curious as to why the backspace is necessary when setting IFS to split on newlines like this:

IFS=$(echo -en "\n\b")

Why can I not just use this (which doesn't work) instead?

IFS=$(echo -en "\n")

I'm on a Linux system that is saving files with Unix line endings. I've converted my file with newlines to hex and it definitely only uses "0a" as the newline character.

I've googled a lot and although many pages document the newline followed by backspace solution, none that I have found explain why the backspace is required.

-David.

user431931
  • 855
  • 1
  • 6
  • 7
  • Could somebody please say where do they see such code? Thank you. – spbnick May 31 '16 at 12:42
  • @spbnick You need to reset the IFS whenever you want to iterate over files, e.g. in `for file in $(ls) ; do echo "$f" ; done`. Without setting the IFS to only newlines, the for loop will echo each space-separated hunk in each file name individually instead of the whole filename. – bleistift2 May 17 '19 at 17:11
  • Thanks @bleistift2, I understand why somebody would want to change IFS and I understand how the code works (my answer is the most popular here). I'm just curious where this particular use of backspace character in IFS comes from, and who came up with it, historically. Hence I'm asking where people have seen this kind of code. – spbnick May 20 '19 at 12:12
  • @spbnick Darn good question, and I'd love to know, too. It *is* a rather mind-bogglingly obscure and misleading invocation. And pointless! If you want to set, say, shell variable `X` to "a", you just say `X="a"`; you don't say `X=$(echo "a")`. And that's before we get to the part that here we're actually setting IFS to something that will additionally split on a backspace, if the input happens to contain one! If the question is "Why?", yours is the right answer, but if the question is "What's a *good* way to set `IFS` to a newline?", I have to commend moddie's answer. – Steve Summit Mar 26 '22 at 11:38

5 Answers5

163

Because as bash manual says regarding command substitution:

Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted.

So, by adding \b you prevent removal of \n.

A cleaner way to do this could be to use $'' quoting, like this:

IFS=$'\n'
spbnick
  • 5,025
  • 1
  • 17
  • 22
  • 1
    `$'...'` is targeted for inclusion in POSIX. [http://mywiki.wooledge.org/Bashism] [http://austingroupbugs.net/view.php?id=249] – go2null Nov 05 '15 at 19:42
  • 2
    While these are supposed to be included in Posix at some point, they currently aren't and this won't work with /bin/sh in many places which links against a posix shell, not bash. – John Eikenberry Dec 26 '15 at 23:40
  • 1
    But when I want to split the array using newline and read command. Only the first item is getting added to the hostArray. But the host has 4 items. `host=$(mysql -S $socket $catalog --skip-column-names -e "select host from mysql.user); echo "Host: $host"; IFS=$'\n';read -ra hostArray <<< "$host"; echo "HostArray:$hostArray" #This prints only 1 item. But there are 4 items;` What is wrong here? – Guna Jul 30 '18 at 12:59
64

I just remembered the easiest way. Tested with bash on debian wheezy.

IFS="
"

no kidding :)

moddie
  • 641
  • 5
  • 2
  • 3
    but it's not a one liner -- specially if you need to restore the old $IFS, as in: http://stackoverflow.com/questions/1406966/linux-shell-script-split-string-put-them-in-an-array-then-loop-through-them#comment1250215_1407098 – juanmirocks Nov 10 '16 at 09:33
  • 1
    Man, you made my day! No kidding and works in posix also. – Aleksey Balenko Jun 08 '18 at 12:16
  • @juanmirocks Why does this solution make it any harder to restore the old `IFS` afterwards? – Steve Summit Mar 26 '22 at 11:40
  • @SteveSummit I've not tested it right now. But I assume you will have to quote the "`oldIFS`" well to avoid any pitfalls. – juanmirocks May 27 '22 at 12:04
14

It's a hack because of the use of echo and command substitution.

prompt> x=$(echo -en "\n")
prompt> echo ${#x}
0
prompt> x=$(echo -en "\n\b")
prompt> echo ${#x}
2

The $() strips trailing newlines and \b prevents \n from being a trailing newline while being highly unlikely to appear in any text. IFS=$'\n' is the better way to set IFS to split on newlines.

Old Pro
  • 24,624
  • 7
  • 58
  • 106
  • 2
    This demonstrates that the `\n\b` formation also includes backspace as an `IFS` delimiter, so I'd consider that broken in some circumstances. – Tom Hale Oct 04 '16 at 14:01
10

The \b char as a suffix of newline \n is added due to removal of the tailing \n in the command substitution $(...). So the \b is used as a suffix to \n, and by that \n is no longer trailing, so it is returned from the the command substitution.

The side effect is, that IFS also will include the \b char as a separator as well, instead of just \n, which really is our sole interest.

If you expect \b may someday appear in the string (why not?), then you may use:

IFS="$(printf '\nx')" && IFS="${IFS%x}";

that returns \n suffixed with x && removes x

now IFS contains only the \n char.

IFS="$(printf '\nx')" && IFS="${IFS%x}";
echo ${#IFS}; # 1

and nothing will break in case of \b, test:

#!/bin/sh

sentence=$(printf "Foo\nBar\tBaz Maz\bTaz");
IFS="$(printf '\nx')" && IFS="${IFS%x}";

for entry in $sentence
do
    printf "Entry: ${entry}.\n";
done

gives two lines (due to one \n):

Entry: Foo.
Entry: Bar      Baz Maz Taz.

as expected.

IFS="$(printf '\nx')" && IFS="${IFS%x}"; using:

IFS="
"

gives the same result, but these two lines must not be indented, and if you accidentally put space or tab or any other white char between " and ", you'll no longer have only the \n char there, but some "bonuses" as well. This bug will be hard to spot, unless you use the option "show all characters" in your editor.

hoijui
  • 3,615
  • 2
  • 33
  • 41
Jimmix
  • 5,644
  • 6
  • 44
  • 71
  • 1
    In POSIX compliant scripts, using `printf` instead of `echo` is mandatory, and using `IFS=$(printf '\nx'); IFS=${IFS%?}` instead of `IFS="``"` is more clear IMHO – Fravadona Feb 06 '22 at 08:27
  • Mine keeps the newline in the second var isn't it supposed to strip it out? – JPM Aug 29 '23 at 21:30
  • @JPM Which example exactly do you mean? – Jimmix Aug 30 '23 at 18:02
-1

Just pressing enter without assigning anything also works. Although, it looks like someone made a mistake and difficult to understand.

IFS=
#This is a line
taiyebur
  • 391
  • 4
  • 13
  • 4
    Doesn't this set $IFS to an empty string? – Marius Gedminas May 03 '19 at 12:19
  • Well, you press enter after the '=' sign for it to work. It worked when i tried. @moddie's answer looks more clear. – taiyebur May 04 '19 at 07:33
  • 2
    This is incorrect. It is functionally equivalent to `IFS=""`, which is not what was asked for. `IFS=` creates a zero-length string, whereas `IFS=$'\n'` creates a string of length 1 character. You can verify with `echo ${#IFS}`. – Rich Jun 25 '21 at 20:59