In Perl, how to remove ^M from a file?

Question

I have a script that is appending new fields to an existing CSV, however ^M characters are appearing at the end of the old lines so the new fields end up on a new row instead of the same one. How do I remove ^M characters from a CSV file using Perl?

Use `binmode(STDIN, ":crlf")` or `PERLIO=:unix:crlf` (see [http://stackoverflow.com/a/21320709/424632]). — musiphil, Jan 23 '14 at 22:35

score 52 · Answer 1 · answered Mar 16 '09 at 14:51

52

^M is carriage return. You can do this:

$str =~ s/\r//g

answered Mar 16 '09 at 14:51

Can Berk Güder

109,922
25
130
137

score 28 · Answer 2 · answered Mar 16 '09 at 16:36

28

Or a 1-liner:

perl -p -i -e 's/\r\n$/\n/g' file1.txt file2.txt ... filen.txt

answered Mar 16 '09 at 16:36

JDrago

2,079
14
15

3

It's so easy to remember this one as Perl Pie. – dreamlax Mar 16 '09 at 19:02
On windows passing *.txt with this command does not work. It gives: Can't open *.txt: Invalid argument. Anyone? – mgouin Dec 10 '15 at 19:26
No need for global 'g' as '$' matches only end of line. – Mohammad Azim Mar 20 '17 at 12:07

score 15 · Accepted Answer · edited Nov 30 '15 at 08:37

15

You found out you can also do this:

$line=~ tr/\015//d;

edited Nov 30 '15 at 08:37

serenesat

4,611
10
37
53

answered Mar 16 '09 at 17:36

Alex Wong

761
3
9
15

1

not as readable as `\r` - anyone looking at that (or yourself in a year's time) would be glad of a comment stating what it does – plusplus Jun 21 '11 at 08:12

score 8 · Answer 4 · edited Dec 07 '17 at 00:20

8

Slightly unrelated, but to remove ^M from the command line using Perl, do this:

perl -p -i -e "s/\r\n/\n/g" file.name

edited Dec 07 '17 at 00:20

Peter Mortensen

30,738
21
105
131

answered Mar 16 '09 at 17:45

Roy Rico

3,683
6
35
36

score 6 · Answer 5 · answered Aug 05 '13 at 15:34

6

I prefer a more general solution that will work with either DOS or Unix input. Assuming the input is from STDIN:

while (defined(my $ln = <>))
  {
    chomp($ln);
    chop($ln) if ($ln =~ m/\r$/);

    # filter and write
  }

answered Aug 05 '13 at 15:34

KillerRabbit

173
1
8

score 3 · Answer 6 · edited Dec 07 '17 at 00:21

3

This one liner replaces all the ^M characters:

dos2unix <file-name>

You can call this from inside Perl or directly on your Unix prompt.

edited Dec 07 '17 at 00:21

Peter Mortensen

30,738
21
105
131

answered Aug 03 '12 at 23:04

Akhil

31
1

spoulson · Answer 7 · 2009-03-16T15:23:16.827

2

To convert DOS style to UNIX style line endings:

for ($line in <FILEHANDLE>) {
   $line =~ s/\r\n$/\n/;
}

Or, to remove UNIX and/or DOS style line endings:

for ($line in <FILEHANDLE>) {
   $line =~ s/\r?\n$//;
}

edited Mar 16 '09 at 15:23

answered Mar 16 '09 at 14:51

spoulson

21,335
15
77
102

wouldn't that remove the newlines, too? – Can Berk Güder Mar 16 '09 at 14:54
I guess that depends on your goal. I edited to show both strategies. – spoulson Mar 16 '09 at 15:23

score 1 · Answer 8 · edited Dec 07 '17 at 00:21

1

This is what solved my problem. ^M is a carriage return, and it can be easily avoided in a Perl script.

while(<INPUTFILE>)
{
     chomp;
     chop($_) if ($_ =~ m/\r$/);
}

edited Dec 07 '17 at 00:21

Peter Mortensen

30,738
21
105
131

answered Mar 17 '16 at 06:57

user3274263

41
1
11

Does that remove ^M from a CSV file? Changing the input file? Does it create some output file that will not have them? – Peter Mortensen Dec 07 '17 at 00:23

score 0 · Answer 9 · answered Sep 18 '16 at 04:28

Little script I have for that. A modification of it helped to filter out some other non-printable characters in cross-platform legacy files.

#!/usr/bin/perl
# run this as
# convert_dos2unix.pl < input_file > output_file
undef $/;
$_ = <>;
s/\r//ge;
print;

score 0 · Answer 10 · edited Dec 17 '21 at 21:12

0

perl command to convert dos line ending to unix line ending with backup of the original file:

perl -pi.bak -e 's/\r\n/\n/g' filename

This command generates filename with unix line ending and leaves the original file as filename.bak.

edited Dec 17 '21 at 21:12

Brian Tompsett - 汤莱恩

5,753
72
57
129

answered Dec 17 '21 at 18:32

Jeongtae Kim

1

score -1 · Answer 11 · edited Dec 07 '17 at 00:20

-1

In vi hit :.

Then s/Control-VControl-M//g.

Control-V Control-M are obviously those keys. Don't spell it out.

edited Dec 07 '17 at 00:20

Peter Mortensen

30,738
21
105
131

answered Mar 16 '09 at 17:45

1

It's a bad idea to include non-printing characters like carriage return verbatim in source code like this. Far better to use the \r escape that is (a) easy to see and (b) won't get lost if the source is reformatted. – Denis Howe Dec 02 '15 at 17:16

In Perl, how to remove ^M from a file?

11 Answers11

Linked