Fix newlines when writing UTF-8 to Text file in python

Question

I'm at my wits end on this one. I need to write some Chinese characters to a text file. The following method works however the newlines get stripped so the resulting file is just one super long string.

I tried inserting every known unicode line break that I know of and nothing. Any help is greatly appreciated. Here is snippet:

import codecs   
file_object = codecs.open( 'textfile.txt', "w", "utf-8" )
xmlRaw = (data to be written to text file )    
newxml = xmlRaw.split('\n')
for n in newxml:
    file_object.write(n+(u'2424'))# where \u2424 is unicode line break

What do you mean by "converted to UTF-8"? What form is it initially? (If it's ASCII text, it's already UTF-8.) — user2357112, Aug 09 '13 at 22:49
its data parsed from an XML file that includes chinese characters. I received character encoding errors when writing to a txt file however it outputted to the console fine. Using this method I can save the Chinese characters, however newlines are dropped — Chris Hall, Aug 09 '13 at 22:52
\u2424 is not an actual newline, it's the "symbol for newline"; in text it will usually be rendered as an "n" and an "l" next to each other, but it will not actually break lines. — timotimo, Aug 10 '13 at 00:10
In python, unicode is actually a standalone type, and one can cast str into unicode, see: http://docs.python.org/2/howto/unicode.html#the-unicode-type — Patrick the Cat, Aug 10 '13 at 00:47

rtxndr · Answer 1 · 2013-08-09T23:30:21.580

4

If you use python 2, then use u"\n" to append newline, and encode internal unicode format to utf when you write it to file: file_object.write((n+u"\n").encode("utf")) Ensure n is of type unicode inside your loop.

edited Aug 09 '13 at 23:30

answered Aug 09 '13 at 22:58

rtxndr

872
1
9
20

serving this error:UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 21: ordinal not in range(128) – Chris Hall Aug 09 '13 at 23:06
what is `type(newxml)` ? – rtxndr Aug 09 '13 at 23:08
put `print type(newxml)` and check it. – rtxndr Aug 09 '13 at 23:13
I checked and its a list – Chris Hall Aug 09 '13 at 23:15
oops I meant `n` inside loop – rtxndr Aug 09 '13 at 23:16
Copy all your source to pastebin or somewhere else and give me link. And tell me python version(`python -V`). – rtxndr Aug 09 '13 at 23:25

score 0 · Answer 2 · answered Dec 23 '18 at 09:48

I had the same problem to the same effect (wit's end and all). In my case, it was not an encoding issue, but the need to replace every '\n' with '\r\n', which led to better understanding the difference between line-feeds and carriage returns, and the fact that Windows editors often require \r\n for line breaks: 12747722

score 0 · Answer 3 · answered Mar 10 '19 at 19:13

The easiest way to do that is using the combination of "\r\n" as marc_a said.

So, your code should look like this:

import codecs   
file_object = codecs.open( 'textfile.txt', "w", "utf-8" )
xmlRaw = (data to be written to text file )    
newxml = xmlRaw.split('\n')
for n in newxml:
    file_object.write(n+u"\r\n")

Fix newlines when writing UTF-8 to Text file in python

3 Answers3