sed behaving strangely when replacing line ends in WSL

Question

I am trying to use sed to add some text, at every line end, for all .txt files in a directory. This is the exact command I use: find . -name "*.txt" -exec sed -i 's/$/:orig/' {} +

Expected:

https://pbs.twimg.com/media/EUr539_UMAAFqZM.jpg:orig
https://pbs.twimg.com/media/ENTrymcUwAAnd6_.jpg:orig
https://pbs.twimg.com/media/EIzzcrFUYAAgfUo.jpg:orig

That is also what I actually get when I run it in my laptop with Linux Mint 19.2. But when I try it in my Windows PC, running sed through Ubuntu in WSL, what I get is this:

https://pbs.twimg.com/media/EUr539_UMAAFqZM.jpg
:orig
https://pbs.twimg.com/media/ENTrymcUwAAnd6_.jpg
:orig
https://pbs.twimg.com/media/EIzzcrFUYAAgfUo.jpg:orig

If I cat the files in question while still in the Ubuntu terminal, what's displayed is more like this (there's some weird whitespace that makes it look like columns in SO, but generally they all look pretty chaotic):

:orig://pbs.twimg.com/media/EUr539_UMAAFqZM.jpg                                                                         :orig://pbs.twimg.com/media/ENTrymcUwAAnd6_.jpg                                                                         https://pbs.twimg.com/media/EIzzcrFUYAAgfUo.jpg:orig

I understand Windows and Linux text is formatted differently and that line ends in particular are problematic, though I am uncertain if that is of any importance here.

Can anyone shed light on this behavior? How can I get the command to behave consistently?

see also: [Why does my tool output overwrite itself and how do I fix it?](https://stackoverflow.com/questions/45772525/why-does-my-tool-output-overwrite-itself-and-how-do-i-fix-it) — Sundeep, Apr 05 '20 at 03:34

paxdiablo · Accepted Answer · 2020-04-05T02:30:10.240

The problem is that your files end in CRLF but the WSL sed uses just LF and the line end. You can get around this with a three step process, if you know it's a CRLF-style file:

get rid of the CR;
do your change;
put the CR back.

That would go something like: sed -i -e 's/\r$//' -e 's/$/:orig/' -e 's/$/\r/'.

However, that this won't work on UNIX-style files since the first substitution will do nothing but the third will put a CR character at the end of each line, even though it wasn't there originally. If you want something that will work on both types of files, this should do it:

sed -E 's/(\r)?$/:orig\1/'

This captures the optional CR at the end of the line and puts it back in the substitution (if it's not in the original line, it won't put it back).

sed behaving strangely when replacing line ends in WSL

1 Answers1