Preserve line endings

Question

I run sed to do some substitution on windows and I noticed that it automatically converts line endings to Unix (\n). Is there an option to tell sed to use Windows line endings (\r\n) or even better to preserve the line endings from the file?

Note: I use sed from unxutils: http://unxutils.sourceforge.net/

I even got this far and it still didn't work `LC_ALL=C perl -i -e 'binmode $STDIN;undef $/;$_=<>;s|http://911coned.com|https://911coned.com|gm;print' education.html` — William Entriken, Mar 17 '17 at 21:01
So actually the command above does work and I just discovered a bug in the `git diff` program. — William Entriken, Mar 17 '17 at 21:13

score 171 · Accepted Answer · edited Oct 12 '15 at 17:35

171

You can use the -b option for sed to have it treat the file as binary. This will fix the problem with cygwin's sed on Windows.

Example: sed -b 's/foo/bar/'

If you wish to match the end of the line, remember to match, capture and copy the optional carriage return.

Example: sed -b 's/foo$\r\?$$/bar\1/'

From the sed man page:

-b --binary
This option is available on every platform, but is only effective where the operating system makes a distinction between text files and binary files. When such a distinction is made—as is the case for MS-DOS, Windows, Cygwin—text files are composed of lines separated by a carriage return and a line feed character, and sed does not see the ending CR. When this option is specified, sed will open input files in binary mode, thus not requesting this special processing and considering lines to end at a line feed.`

edited Oct 12 '15 at 17:35

Joseph Thomson

9,888
1
34
38

answered Jul 16 '12 at 16:38

Shlomo

1,784
1
11
3

6

Note that this does not work with `sed -i` on cygwin (for me), but you can work around that. Thanks for the update -- the other answers were the last word on this subject for a while. – harpo Nov 19 '12 at 22:57
Note, this option is not available with sed on Mac. – Senthil Kumaran Dec 26 '13 at 20:58
Doesn't work for me in Ubuntu 12.04. Adding --binary gave the exact same output (removed Windows carriage return on replaced lines). – John Jan 17 '14 at 20:07
27

Works for me even with `sed -i`: it is just important how to type it. While `sed -bi` and `sed -i -b` work, `sed -ib` does **not** work: see the man page for why (uses the `b` as the suffix for the backup copy). – Olaf Mandel May 04 '15 at 13:34
2

Use: `sed -bi 's/foo/bar/'` – Kunal B. Sep 14 '15 at 11:55
3

Does not work for me in Windows cygwin. On the lines that sed made the change, the line endings is Unixy. The rest of lines have Windowish line-endings. Thus my file got a mixed of lines with different line endings. – daparic Mar 24 '17 at 23:21
I tried to [remove BOM](https://unix.stackexchange.com/questions/381230/how-can-i-remove-the-bom-from-a-utf-8-file) with `sed -bi "1s/^\xEF\xBB\xBF//"` using gnuwin32 sed 4.2.1 but it corrupts each line ending from `\r\n` to `\r\r\n`. ( Look at @buckley's answer for a workaround. – Vadzim Dec 14 '17 at 17:29
Found a version not corrupted with corruption. Take a look at [my answer](https://stackoverflow.com/questions/4652652/preserve-line-endings/47834520#47834520). – Vadzim Dec 15 '17 at 14:40

SiegeX · Answer 2 · 2011-01-10T23:38:07.883

10

You could try to sub the \n for \r\n at the end of your existing script like so:

sed 's/foo/bar/;s/$/\r/'

or perhaps

 sed -e 's/foo/bar/' -e 's/$/\r/'

If neither of the above two work, you'll have to consult the specific man page for your version of sed to see if such an option exists. Note that the *nix versions of sed do not alter the line terminators without being told to do so.

Another alternative is to use the cygwin version of sed which shouldn't have this undesirable behavior.

edited Jan 10 '11 at 23:38

answered Jan 10 '11 at 23:24

SiegeX

135,741
24
144
154

16

The cygwin version does have this undesirable behavior. – harpo Feb 09 '11 at 06:20
3

If file contains both \n (`0x0A`) and \r\n (`0x0D 0x0A`) - this proposed solution (of always re-injecting the \r) breaks it. – Vlad Sep 18 '15 at 13:23
This works for me using MSYS2/MinGW. Thank you @SiegeX. – AntumDeluge Feb 05 '19 at 00:35

score 7 · Answer 3 · answered Nov 28 '11 at 19:34

7

Alternatively, (the cygwin version of) perl -pe doesn't seem to have this problem.

answered Nov 28 '11 at 19:34

seeker

1,136
1
13
16

sed on MacOS does not have the -b option and has similar issues as described in the original question. The perl alternative doesn't have this issue, so thanks for your suggestion. `sed -i -e 's/]*\/>//g' *.xml` replaces line endings by '\n' `perl -i -p -e 's/]*\/>//g' *.xml` preserves the original line endings – Guruniverse Apr 08 '17 at 11:02

score 2 · Answer 4 · answered Oct 16 '17 at 13:50

2

Gnuwin can be suppressed to mess up the newlines (win->unix) if you only specify the -b switch and redirect. Using the -i (inline) switch will mess it up.

E.g. sed.exe -b "s/\xFF\xFE//" c:\temp\in.csv > c:\temp\out.csv

answered Oct 16 '17 at 13:50

buckley

13,690
3
53
61

1

See a version with working `-i` mode in [my answer](https://stackoverflow.com/questions/4652652/preserve-line-endings/47834520#47834520). – Vadzim Dec 15 '17 at 14:38

score 2 · Answer 5 · answered Dec 15 '17 at 14:37

I've found that sed-4.4.exe from https://github.com/mbuilov/sed-windows is pure win as it

uses windows CRLF line endings in default mode
preserves original line endings in -b mode
works correctly with in-place -i mode
also offers -z mode with \0 delimeters instead of \n which may be handy sometimes too

See also list of sed options and list of all windows sed ports.

Note that gnuwin32 sed 4.2.1 does corrupt line endings in -bi mode and doesn't have -z mode at all.

Preserve line endings

5 Answers5

Linked