Why is \r\n being converted to \n when a string is saved to a file?

Question

The string is originating as a return value from:

> msg = imap.uid_fetch(uid, ["RFC822"])[0].attr["RFC822"]

In the console if I type msg, a long string is displayed with double quotes and \r\n separating each line:

> msg
"Delivered-To: email@test.com\r\nReceived: by xx.xx.xx.xx with SMTP id;\r\n"

If I match part of it with a regex, the return value has \r\n:

> msg[/Delivered-To:.*?\s+Received:/i]
=> "Delivered-To: email@test.com\r\nReceived:"

If I save the string to a file, read it back in and match it with the same regex, I get \n instead of \r\n:

> File.write('test.txt', msg)
> str = File.read('test.txt')
> str[/Delivered-To:.*?\s+Received:/i]
=> "Delivered-To: email@test.com\nReceived:"

Is \r\n being converted to \n when the string is saved to a file? Is there a way to save the string to a file, read it back in without the line endings being modified?

may be its due to the operating system? see this: http://stackoverflow.com/q/27323188/3286583 — Devesh Khandelwal, Dec 14 '15 at 19:15

score 10 · Accepted Answer · edited May 23 '17 at 12:31

This is covered in the IO.new documentation:

The following modes must be used separately, and along with one or more of the modes seen above.

"b"  Binary file mode
     Suppresses EOL <-> CRLF conversion on Windows. And
     sets external encoding to ASCII-8BIT unless explicitly
     specified.

"t"  Text file mode

In other words, Ruby, like many other languages, senses the OS it's on and will automatically translate line-ends between "\r\n" <-> "\n" when reading/writing a file in text mode. Use binary mode to avoid translation.

str = File.read('test.txt')

A better practice would be to read the file using foreach, which negates the need to even care about line-endings; You'll get each line separately. An alternate is to use readlines, however it uses slurping which can be very costly on large files.

Also, if you're processing mail files, I'd strongly recommend using something written to do so rather than write your own. The Mail gem is one such package that's pre-built and well tested.

Why is \r\n being converted to \n when a string is saved to a file?

1 Answers1