95

This stackoverflow question has an answer to replace newlines with sed, using the format sed ':a;N;$!ba;s/\n/ /g'.

This works, but not for special characters like \r, \n, etc.

What I'm trying to do is to replace the newline character by a literal \n. Tried

sed ':a;N;$!ba;s/\n/\\n/g'

and

sed ':a;N;$!ba;s/\n/\\\n/g'

also

sed ":a;N;$!ba;s/\n/'\'n/g"

but all to no avail. Sed keeps replacing the newline character.... with a newline character.

Thoughts?

Edited after first answer:

For the sake of completeness the commands run are :

PostContent=cat $TextTable | sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g'

Where TextTable is a variable linking to a text file containing a JSON output in the following format :

{"posts":[{"title":"mysupertest","slug":"bi-test","markdown":"##TEST
First things first !
To TEST this TEST TEST, click the download button below.
If you need more information about the TEST TEST, you can  read the Table of Contents below.

<a href='/assets/TEST.pdf' style='border-radius:5px; padding: 4px 15px; background-color:#008CBA; color:white; text-decoration:none; float:right;' download> Download </a>


##TEST OF TEST


###TEST TEST PLATFORM TEST GUIDE
WaTESTve TEST SetupTEST
TESTTEST
TESTTESTETESTTETSTTEST
TESTTESTTTETST
TESTTES
TESTTESTESSTSTESTESTTES
TEST","image":"http://localhost:3000/myimage.jpg","featured":false,"page":false,"status":"draft","language":"en_US","meta_title":null,"meta_description":null,"author":"4","publishedBy":null,"tags":[{"uuid":"ember2034","name":"implementation guides","slug":null,"description":null,"meta_title":null,"meta_description":null,"image":null,"visibility":"public"}]}]}
anubhava
  • 761,203
  • 64
  • 569
  • 643
Parze
  • 953
  • 1
  • 6
  • 6
  • 3
    When you find yourself writing/using commands like `sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g'`, doesn't it feel like it's getting just a wee bit arcane? If you post some concise, testable sample input and expected output I'm sure someone can show you how to solve your problem in a clear, efficient, portable and easily maintainable way! – Ed Morton Jul 30 '16 at 14:03

6 Answers6

140

Is this what you're trying to do?

$ cat file
a
b
c

$ awk '{printf "%s\\n", $0}' file
a\nb\nc\n$

or even:

$ awk -v ORS='\\n' '1' file
a\nb\nc\n$

Run dos2unix on the input file first to strip the \rs if you like, or use -v RS='\r?\n' with GNU awk or do sub(/\r$/,""); before the printf or any other of a dozen or so clear, simple ways to handle it.

sed is for simple substitutions on individual lines, that is all. For anything else you should be using awk.

Ed Morton
  • 188,023
  • 17
  • 78
  • 185
  • 30
    This is perfect for, for example, preparing an SSL cert to be added single-line to a json data bag. Worked on MacOS as well. Upvoted. – EugeneRomero Apr 13 '17 at 22:18
  • 3
    this works for me on macOS, I was trying to achieve the same to convert key/pem file to single liner for a json databag.. – Chen Xie Apr 09 '18 at 21:08
  • 2
    This would append an additional `\n` at the end. – thetaprime Sep 09 '21 at 05:18
  • 1
    @thetaprime what is "this" in your comment? If it's either of the scripts in my answer - no, it replaces each newline in the input with `\n` as requested by the OP and as you can see in my answer. – Ed Morton Sep 09 '21 at 12:33
  • @EdMorton I think @thetaprime got confused as I initially did because of the \r. After re-reading your post the following worked for me: `awk '{gsub(/\r/$,""); ORS="\\n"}1' file`. Sharing this in case someone find it useful. – luissquall May 08 '23 at 19:07
54

This should work with both LF or CR-LF line endings:

sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g' file
anubhava
  • 761,203
  • 64
  • 569
  • 643
  • 2
    Thanks for answering. Unfortunately I have the same result using this command. I have updated the question with more details about the process. – Parze Jul 30 '16 at 10:22
  • 2
    Can you post output of `cat -vte file.json` in question as well? – anubhava Jul 30 '16 at 10:35
  • 3
    Nevermind, I'm an idiot. Echo interpreted the newline character as a newline. Thank you for your answer. – Parze Jul 30 '16 at 10:37
  • [Here is a working demo of above command](http://ideone.com/plzyCP) - check output under **stdout** section. – anubhava Jul 30 '16 at 10:37
  • Say, if I wanted to replace it with
    for example, I could adapt your solution to sed -E ':a;N;$!ba;s/\r{0,1}\n/\
    /g', if I'm not mistaken. Could you explain how your code works ? I don't get the first part.
    – Parze Jul 30 '16 at 11:14
  • You can use `sed -E ':a;N;$!ba;s/\r{0,1}\n/
    /g' file` for using `
    ` in replacement. `\r{0,1}\n` matches `\r\n` or `\n` in regex
    – anubhava Jul 30 '16 at 11:23
  • 2
    You can also replace `{0,1}` by `?`, that seems more readable to me. – sylbru Jan 27 '17 at 15:00
  • By the way, I wondered about this `-E` option, not mentioned in `man sed`, and it turns out it's an alias for the `-r` option, which means “use extended regular expressions in the script.” – sylbru Jan 27 '17 at 15:02
  • @Niavlys yes that's right and `-E` is more portable option that `-r` across various *nix flavors. – anubhava Jan 27 '17 at 15:17
  • This didn't work for me. It appeared to have no effect at all. – user124384 Aug 07 '18 at 21:22
  • @user124384: Try awk solution by Ed – anubhava Aug 07 '18 at 21:26
  • 3
    @anubhava: I've been using `cat` for a quarter century now, but never even bothered to look at the man page (DOH!); I'll keep `-vte` in mind for the future - thanks for that! :-) – ssc Jun 24 '20 at 08:43
  • `:a` sets a label named `a`. `$!ba` jumps back to label `a` for anything but last line. – anubhava Nov 05 '21 at 15:44
31

You could do this using sed and tr:

sed 's/$/\\n/' file | tr -d '\n'

However this will add an extra \n at the end.

mvr
  • 1,732
  • 16
  • 11
  • 6
    The last `\n` can be easily worked around with `$!`: `sed '$!s/$/\\n/' file | tr -d '\n'`. Otherwise, awesome answer :) – x-yuri Nov 14 '19 at 10:28
14

With the -zoption you can do

sed -z 's/\n/\\n/g' file

or

sed -z "s/\n/\\\n/g" file
Walter A
  • 19,067
  • 2
  • 23
  • 43
  • 3
    `-z` seems to be available in GNU sed only, not in BSD sed; under macOS, the usual `brew install gnu-sed` option applies – ssc Jun 24 '20 at 08:40
  • @ssc I tried to keep the answer simple, `printf "%s" $(sed 's/$/\\n/' file)` is just another alternative. – Walter A Jun 24 '20 at 21:49
  • 1
    @mvr's answer (see [above](https://stackoverflow.com/a/43967678/3814775)) is better than this one, since this one has two limitations, 1 as @ssc mentioned, the `-z` is GNU-specific option; 2 it can't avoid the extra `\n` at the end while @mvr's answer can. However, I still upvoted this answer, because it is a pretty cool option! Good to learn the `-z` option. Thank you! – Bruce Nov 15 '21 at 21:30
5

In case it helps anyone, I was searching for the opposite of this question: to replace literal '\'n in a string with newline. I managed to solve it with sed like this:

_s="foo\nbar\n"
echo $_s | sed 's/\\n/\n/g'
RSX
  • 376
  • 4
  • 14
  • 1
    That echo would do different things on different systems (use `printf '%s\n' "$_s"` instead) and not quoting the variable isn't safe, see https://mywiki.wooledge.org/Quotes. In bash at least you don't need the pipe and sed at all, though, you could just do `printf '%s\n' "${_s//\\n/$'\n'}"` – Ed Morton Feb 08 '20 at 14:06
-2

Here is little python script for replacing the '\r\n' with '\r' in directory in a recursive way import os import sys

if len(sys.argv) < 2:
    print("Wrong arguments. Expected path to directory as arg 1.")
    exit(1)

path = sys.argv[1]


def RecOpOnDir(path, op) :
    for f in os.listdir(path):
        full_f = path + "/" + f
        if os.path.isdir(full_f):
            RecOpOnDir(full_f, op)
        else:
            try:
                op(full_f)
            except Exception as ex:
                print("Exception during proc '", full_f, "' Exception:", ex)

file_counter = 0

def WinEndingToUnix(path_to_file):
    global file_counter
    file_counter += 1
    file_strs = []
    with open(path_to_file) as f:
        for line in f:
            file_strs.append(line.replace(r"\r\n", r"\n"))

    with open(path_to_file, "w") as fw:
        fw.writelines(l for l in file_strs)

try:
    RecOpOnDir(path, WinEndingToUnix)
    print("Completed.", file_counter, "files was reformed")
except Exception as ex:
    print("Exception occured: ", ex)
    exit(1)
voltento
  • 833
  • 10
  • 26