Cross-platform newline confusion

Question

For some reason, my write-to-textfile function stopped working all of a sudden.

void write_data(char* filename, char* writethis)
{
    ofstream myfile;
    myfile.open (filename, std::ios_base::app);
    myfile << endl << writethis;
    myfile.close();
}

The function was called from a loop, so basically it started with an empty line and appended all the following "writethis" lines on a new line.

Then all of a sudden, no more newlines. All text was appended on one single line. So I did some digging and I came across this:

Windows = CR LF
Linux = LF
MAC < 0SX = CR

So I changed the line to

myfile << "\r\n" << writethis;

And it worked again. But now I'm confused. I am coding on linux but I am reading the textfiles created with the program out on windows after transferring them with filezilla. Now which part of this caused the lines in the textfile to appear as one line?

I was pretty sure "endl" worked just fine for linux so now I'm thinking windows messed the file up after transferring them with filezilla? Messing up the way the text file is written to (and read out) will guarantee my program to break, so if someone can explain this I'd appreciate it.

I also don't recall what I changed in my program to cause this to break, because it was working just fine earlier. The only thing I added was threading.

Edit: I have tried swapping the transfer mode from ASCII / Binary (even removed the force-ASCII-for-txt-extension), but it makes no differences. The newlines appear in linux, but not on windows. fz-messup

How odd.

possible duplicate of [What is the difference between \r and \n?](http://stackoverflow.com/questions/1279779/what-is-the-difference-between-r-and-n) Also consider http://stackoverflow.com/q/728312/10468 — DarenW, Nov 07 '11 at 08:17
I dissagree, I find the questions to be quite different in scope. But I wouldn't object this getting closed, I just wouldn't agree with the reason. — natli, Nov 07 '11 at 08:20
Re-opening and re-closing a file for every line of output is inefficient. Regardless, the function body can be written more simply as `ofstream(filename, std::ios_base::app) << endl << writethis;`. This takes full advantage of RAII. Also, you really should be using `std::string` to represent text items. — Karl Knechtel, Nov 07 '11 at 11:04

score 11 · Accepted Answer · edited Apr 01 '19 at 13:57

11

What happens is that you write the Unix line endings ('\n'), then transfer it to a Windows machine getting a bitwise identical file, then trying to open the file with a viewer that does not understand Unix line endings (Notepad likely).

From my experience on writing portable code:

Standardize on ONE line-ending ('\n', LF) on ALL platforms.
Always open your files in binary, even if you write text.
Let the user who opens the file use a text viewer that understands any line-endings. There are plenty for windows (including Visual Studio, Notepad++, Wordpad and your favorite browser).

Yes, I do think that there is more benefit to everybody to standardize on one thing rather than supporting all of them everywhere. Also I deny the existence of "proper line endings on the proper platform". The fact that Microsoft decided that their native API does not speak UTF-8 or does not understand Unix line endings does not prevent everybody's code from doing that, on Windows. Just make sure not to pass this stuff to WinAPI. Many times you do text processing on your internal data that the system will not ever see, so why the hell do you need to complicate your life by meeting the expectations of those system's internals?

edited Apr 01 '19 at 13:57

yoozer8

7,361
7
58
93

answered Nov 07 '11 at 08:07

Yakov Galka

70,775
16
139
220

1

This certainly eradicates the problem, but it's a bit of a tall order to expect an arbitrary user to be ready for this, and by sidestepping the problem you're not solving it at all. The correct approach is to _fix the file transfers_ so that you have the proper line endings on the proper platform. And the great news is, it's not hard! – Lightness Races in Orbit Nov 07 '11 at 08:37
3

@TomalakGeret'kal: It's impossible, you can't in general determine what file is textual and what is binary. And now imagine two machines, one Linux and one Windows writing to the same file on a NAS. Who is responsible and for what conversion? – Yakov Galka Nov 07 '11 at 08:40
2

@ybungalobill: That's great if you are the only person writting to files. Unfortunately you have to cope with files that are written by other programs (here we definitely have filezilla and probably other text editors that may write the file (in a modified format from what was read)). So really you need to explain in detail what is happening so the the OP can take into account these other programs and compensate appropriately. Though I agree with your technique in the small scale it is not enough information for the general case that the OP will run into. – Martin York Nov 07 '11 at 09:51
@ybungalobill: If the OP doesn't know what file format he has in front of him, there are _much_ bigger problems. – Lightness Races in Orbit Nov 07 '11 at 18:18
@TomalakGeret'kal: You were talking about "an arbitrary user" and "the file transfers", neither are the OP. – Yakov Galka Nov 07 '11 at 18:51

score 6 · Answer 2 · answered Nov 07 '11 at 07:58

6

endl does "work just fine for Linux". Streaming endl streams a \n character and flushes the stream. Always.

However, a file stream in text mode will convert this \n to \r\n at the implementation layer on Windows, and you'll often find line endings being converted as you transfer the file between platforms, too.

This is probably not a C++ problem, and nothing is "broken"; you should probably configure FileZilla to treat your file as text rather than "binary" (a mode in which line endings are not converted). If your file has no name extension like ".txt" then it probably doesn't do this by default.

answered Nov 07 '11 at 07:58

Lightness Races in Orbit

378,754
76
643
1,055

Two other people are suggesting I open as binary, and you suggest I open as text. Neither work, but this only adds to the confusion. – natli Nov 07 '11 at 08:10
1

@natli You haven't tried to do my 3rd bullet, so don't say it doesn't work. It's in fact your only option when you open a file on a shared storage. In the end what matters is what viewer you use, and not what exactly is written. – Yakov Galka Nov 07 '11 at 08:14
The newlines show in Programmer's Notepad but not in Notepad. I find this odd, and it can cause allot of confusion. I thought I did do your third bullet by trying both binary and ASCII ? I don't mind letting this rest but I do find it odd. – natli Nov 07 '11 at 08:18
2

@natli: Nothing's odd. Microsoft Notepad is a joke as a text viewer. It understands nothing but `\r\n`, can't work with large files, etc... So what? Just don't use it. And my 3rd bullet says nothing on binary and ASCII. – Yakov Galka Nov 07 '11 at 08:21
@ybungalobill Alright then, if there's no way to fix it for notepad then I guess that's that. – natli Nov 07 '11 at 08:23
@natli: you can use "\r\n" as your standard encoding if you think Notepad is a necessity. Just use it anywhere then. – johndodo Nov 07 '11 at 08:28
@johndodo: although possible, it's not very good. The problem is that C++ standard *does* prefer `\n` line endings. This is seen in the definition of `endl`, as Tomalak correctly noted. – Yakov Galka Nov 07 '11 at 08:33
@natli: Agreed. But if he wants to use Notepad, he can. Depends on what is more important. :) – johndodo Nov 07 '11 at 08:35
He can still use Notepad, when he gets the encoding conversion right in his file transfer software. None of this has anything to do with C++. – Lightness Races in Orbit Nov 07 '11 at 08:36

score 3 · Answer 3 · answered Nov 07 '11 at 07:58

3

FTP can mess up your files (that is, it converts newlines) if you transfer files as ASCII. Try transfering as BIN (binary).

answered Nov 07 '11 at 07:58

johndodo

17,247
15
96
113

1

Quite the opposite. The problem is that the line endings are _not_ being converted for his platform, where he wants them to be. – Lightness Races in Orbit Nov 07 '11 at 08:00
Depends on what he wants. I always try to do my own newline conversions - automagic usually just adds to the confusion... Write text, transfer as binary, read text. It will be in the same format as it was written. If you then still have problems with newlines, you can tackle a smaller problem. – johndodo Nov 07 '11 at 08:24
1

Actually, ybungalobill summed it up pretty nicely. :) – johndodo Nov 07 '11 at 08:26

score 2 · Answer 4 · edited Nov 07 '11 at 18:19

Internally all applications use '\n' to indicate line termination.

The problem is that the line termination sequence is platform specific for text files (as your research turned up) Note: Text files, this is the default format when you open a file. If you explicitly select binary when opening a file no translation happens when reading/writing.

What this actually means is that the '\n' character is transformed into a platform specific sequence of character when you write it to a file. But also note that this platform specific sequence is converted back to '\n' when the file is read. The problem you are encountering is that you have written the files on one platform and read them on another.

On linux the line termination sequence is LF ('\n'). Thus you write the file and all '\n' are converted into 'LF' characters. You transfer these files to a windows system and now read the file. On windows the line termination sequence is 'CRLF' So the editor that read the file is looking for two characters to convert back to '\n' but does not find these characters. Now it depends on how smart the editor is as to whether you get a single line or multiple lines.

Cross-platform newline confusion

4 Answers4