remove ^M characters from file using sed

Question

I have this line inside a file:

ULNET-PA,client_sgcib,broker_keplersecurities
,KEPLER

I try to get rid of that ^M (carriage return) character so I used:

sed 's/^M//g'

However this does remove everything after ^M:

[root@localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities^M,KEPLER

[root@localhost tmp]# sed 's/^M//g' test
ULNET-PA,client_sgcib,broker_keplersecurities

What I want to obtain is:

[root@localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities,KEPLER

@fedorqui It's not `^` and `M` literally, it's `^M`, IIRC, called a `control character`. — Jite, Oct 16 '13 at 15:25

score 68 · Accepted Answer · edited May 23 '17 at 12:26

68

Use tr:

tr -d '^M' < inputfile

(Note that the ^M character can be input using Ctrl+VCtrl+M)

EDIT: As suggested by Glenn Jackman, if you're using bash, you could also say:

tr -d $'\r' < inputfile

edited May 23 '17 at 12:26

Community

1
1

answered Oct 16 '13 at 14:44

devnull

118,548
33
236
227

3

+1 for the simplest solution. Also note that you can use octal codes in tr, like `tr -d '\012'` to remove newlines for example. And you can use `od -bc inputfile` to browse what codes mean what and are where in a file. – John Zwinck Oct 16 '13 at 14:50
1

@twalberg I gues `dos2unix` would remove CR only from the end of line. The example in the question doesn't make it evident whether it's present only at the end of line. – devnull Oct 16 '13 at 15:02
@devnull I guess maybe it's not universal, but my `dos2unix` has several different options for controlling conversion mode that may be appropriate (in particular the ones for dealing with Mac files, that seem to like to use `^M` line endings). It may not be a perfect solution for the OPs current question, but it's a good thing to know about, hence the "see also...". – twalberg Oct 16 '13 at 15:06
13

If you're using bash, you can write `tr -d $'\r'` – glenn jackman Oct 16 '13 at 15:10
@twalberg => Yes, dos2unix can be made to do this on Mac, see my [answer](http://stackoverflow.com/questions/19406418/remove-m-characters-from-file-using-sed/31517990#31517990), below. – Big Rich Jul 20 '15 at 13:39
@glennjackman May I know what does the dollar sign "$" do there? I did some search, but wasn't able to figure it out. Thanks! – user2008151314 Sep 16 '16 at 18:00
that is [bash ANSI-C quoting](https://www.gnu.org/software/bash/manual/bashref.html#ANSI_002dC-Quoting) – glenn jackman Sep 16 '16 at 20:47

score 24 · Answer 2 · edited Jan 07 '15 at 00:20

24

still the same line:

sed -i 's/^M//g' file

when you type the command, for ^M you type Ctrl+VCtrl+M

actually if you have already opened the file in vim, you can just in vim do:

:%s/^M//g

same, ^M you type Ctrl-V Ctrl-M

edited Jan 07 '15 at 00:20

Yzmir Ramirez

1,281
7
11

answered Oct 16 '13 at 14:47

Kent

189,393
32
233
301

1

Just hold Ctrl and press V then M – user3439968 Dec 04 '14 at 19:41
The OP said he wanted a solution using `sed`. – Yzmir Ramirez Jan 06 '15 at 23:53
Use $ tr -d '\015' file2; mv file2 file1 – 42n4 Feb 04 '16 at 23:35
This worked, please check this link for more variations: https://www.cyberciti.biz/faq/sed-remove-m-and-line-feeds-under-unix-linux-bsd-appleosx/ – Murphy Nov 07 '17 at 19:23

score 12 · Answer 3 · answered Jul 20 '15 at 13:54

12

You can simply use dos2unix which is available in most Unix/Linux systems. However I found the following sed command to be better as it removed ^M where dos2unix couldn't:

sed 's/\r//g' < input.txt >  output.txt

Hope that helps.

Note: ^M is actually carriage return character which is represented in code as \r What dos2unix does is most likely equivalent to:

sed 's/\r\n/\n/g' < input.txt >  output.txt

It doesn't remove \r when it is not immediately followed by \n and replaces both with just \n. This fails with certain types of files like one I just tested with.

answered Jul 20 '15 at 13:54

Angsuman Chakraborty

603
6
12

1

nice explanation!! Thnks!! – Alejandro Teixeira Muñoz Jan 28 '16 at 21:00
1

Indeed, nice explanation. Your approach does seem to work on any instance I've tested where dos2unix failed. – tasslebear May 06 '16 at 15:57

score 7 · Answer 4 · answered Mar 28 '15 at 08:24

7

alias dos2unix="sed -i -e 's/'\"\$(printf '\015')\"'//g' "

Usage:

dos2unix file

answered Mar 28 '15 at 08:24

ling

9,545
4
52
49

how can you use this command in a shell script, without alias? kept getting sed to complain about some syntax....like solution with \015 char vs ^M – bjm88 Apr 16 '15 at 18:19
1

@bjm88 I found this somewhere: `sed 's,'$'\015'',,'` – glormph Oct 21 '16 at 09:38

Chris Koknat · Answer 5 · 2016-05-09T21:39:05.730

3

If Perl is an option:

perl -i -pe 's/\r\n$/\n/g' file

-i makes a .bak version of the input file
\r = carriage return
\n = linefeed
$ = end of line
s/foo/bar/g = globally substitute "foo" with "bar"

edited May 09 '16 at 21:39

answered Nov 02 '15 at 23:41

Chris Koknat

3,305
2
29
30

how do i use the same in perl script. `test.pl` – dhpratik Sep 23 '17 at 21:21
#!/usr/bin/perl open IN, "<", shift; open OUT, ">", shift; while ( ) { s/\r\n$/\n/g; print OUT; } close(IN); close(OUT); – Chris Koknat Sep 25 '17 at 19:12
test.pl infile outfile – Chris Koknat Sep 25 '17 at 19:12
Note that the test.pl solution in the comments will NOT create a backup file. Instead, it will write a separate file. – Chris Koknat Sep 25 '17 at 19:18

score 2 · Answer 6 · answered Nov 01 '16 at 10:43

2

In awk:

sub(/\r/,"")

If it is in the end of record, sub(/\r/,"",$NF) should suffice. No need to scan the whole record.

answered Nov 01 '16 at 10:43

James Brown

36,089
7
43
59

score 2 · Answer 7 · edited Aug 03 '20 at 20:51

2

This is the better way to achieve

tr -d '\015' < inputfile_name > outputfile_name

Later rename the file to original file name.

edited Aug 03 '20 at 20:51

RavinderSingh13

130,504
14
57
93

answered Jul 12 '20 at 06:53

Mantu

133
1
5

score 1 · Answer 8 · edited May 23 '17 at 10:31

I agree with @twalberg (see accepted answer comments, above), dos2unix on Mac OSX covers this, quoting man dos2unix:

To run in Mac mode use the command-line option "-c mac" or use the commands "mac2unix" or "unix2mac"

I settled on 'mac2unix', which got rid of my less-cmd-visible '^M' entries, introduced by an Apple 'Messages' transfer of a bash script between 2 Yosemite (OSX 10.10) Macs!

I installed 'dos2unix', trivially, on Mac OSX using the popular Homebrew package installer, I highly recommend it and it's companion command, Cask.

score 1 · Answer 9 · edited Nov 22 '19 at 12:54

1

This is clean and simple and it works:

sed -i 's/\r//g' file

where \r of course is the equivalent for ^M.

edited Nov 22 '19 at 12:54

Krishna Vyas

1,009
8
25

answered Nov 22 '19 at 11:00

MAX TAM

19
1

Not all `sed` variants recognize the escape `\r`. In original Unix, it would simply mean a regular `r` character with an unnecessary escape which is simply ignored. This is still the behavior on BSD / macOS. – tripleee Oct 06 '20 at 10:37

score 0 · Answer 10 · answered Sep 19 '22 at 15:48

0

Simply run the following command:

sed -i -e 's/\r$//' input.file

I verified this as valid in Mac OSX Monterey.

answered Sep 19 '22 at 15:48

Sujay Kumar

520
8
19

score 0 · Answer 11 · answered Sep 21 '22 at 10:10

0

remove any \r :

nawk 'NF+=OFS=_' FS='\r'
gawk   3  ORS=   RS='\r'

remove end of line \r :

mawk2 8 RS='\r?\n'
mawk -F'\r$' NF=1

answered Sep 21 '22 at 10:10

RARE Kpop Manifesto

2,453
3
11

remove ^M characters from file using sed

11 Answers11

Linked

Related