2

Problem: My Django app is introducing a CR before each CRLF when writing to a file a string which has been read in via render_to_string. In my template files, I used CRLFs, and the Django processing app writes a file which adds a CR before each CRLF. See below for code details.

However: this only happens with my app which is on my clients' Windows Server 12R2 VM which is located in France. My physical Windows 10 laptop, my ubuntu instances, and my AWS Windows Server 12 instance (which I use for testing before installing on my client's machine) do not add the CRs.

Caveat: For security reasons, I can only access my client's VM via remote desktop, so I need to set an appointment with the IT group in order to explore and debug the problem. And, that is a 9 hour time difference and takes his valuable time. Thus, I need to re-create this problem on my AWS instance so that I can try to debug my code. And, when I say it introduces a CR instead of CRLF, then I might be remembering it wrong -- it could be that my code introduces a LF instead of a CRLF. I have been using Notepad++ and "showing all characters" in order to see the CR (or LF) and CRLF in the generated files. I think that Notepad++ said my faulty file is "Macintosh" while my starting template file is "Dos\Windows."

So what: Well, the generated file is an R script. And, windows R dislikes CR as newlines by giving an error after reading the first line. If I take the generated R script and open it in Notepad++ and then change the newlines (via the lower right hand of the status bar) to Dos\Windows, then my Rscript will work fine.

A few more details: Here is some of the code so you can see exactly how I am reading/rendering the template and how I am saving it to a file. There are no non-ASCII chars in my template nor in the template dict, so I don't suspect it is a unicode issue (I am using python 2):

rstring = render_to_string('helpers.R') + "\n" + render_to_string('Rcalc-template.R', {'debug':1})
rscript_file = os.path.join(tmpdir, 'Rcalc-rendered.R')
fh = open(rscript_file, mode="w")
with fh:
    fh.write(rstring)
subprocess.call(['Rscript', rscript_file], 
                universal_newlines=True, 
                stdout=log_filename, 
                stderr=subprocess.STDOUT, 
                shell=False)

What I need from you: if anyone has an idea about how I can get my AWS Windows 2012R2 instance to behave like the Windows 2012R2 VM in France, then that would be super helpful. I tried changing the language of my AWS instance to French, but that didn't do it, i.e. my app generated files with CRLFs only. I'm not really sure where to go from here. Thanks in advance!

brfox
  • 323
  • 3
  • 14
  • Maybe I need to write the file back in binary mode instead so that python doesn't adjust the newlines? http://stackoverflow.com/questions/41369772/how-to-change-end-of-line-conventions – brfox Jan 07 '17 at 16:51
  • or another reference: http://stackoverflow.com/questions/2536545/how-to-write-unix-end-of-line-characters-in-windows-using-python – brfox Jan 07 '17 at 17:43
  • On Windows, writing `\n` to a file translates to a `\r\n` unless you specify binary mode when opening the file. Use `'wb'` if you are writing the `\r\n` explicitly or you will get `\r\r\n` instead. – Mark Tolonen Jan 07 '17 at 22:49
  • That makes sense, I am not currently using 'wb' -- do you have any idea why it would behave differently on two different Windows machines? Both are Python 2.7 on Win Server 2012R2. – brfox Jan 08 '17 at 04:37
  • I don't know why that would happen. – Mark Tolonen Jan 08 '17 at 04:44
  • @Mark, thanks for making some comments, I appreciate your time. It turns out that I should be embarassed because there was a Django version difference. My deployed server had django 1.7.1 and my test version had 1.8.17. So, I wonder if the File update for universal newlines is what made the fix: https://docs.djangoproject.com/en/1.10/releases/1.8/ – brfox Jan 10 '17 at 03:50

1 Answers1

0

Django versions I had a chance to remote into the server where my code was deployed for a few minutes, and there was a Django version difference! My deployed server had django 1.7.1 and my test version had 1.8.17. So, I went to my test AWS server and moved the Django version back to 1.7.1 and sure enough, the error was reproducible! So, it wasn't a Windows or localization or language issue...

In the release notes, it mentioned support in File for universal newlines: https://docs.djangoproject.com/en/1.10/releases/1.8

I think this is the ticket and commit which fixed it?

https://code.djangoproject.com/ticket/8149 https://github.com/django/django/commit/eb4f6de980c5148ba48d4ed67f31cca27dd132a8

I was also able to fix the problem by replacing all '\r' with '\n':

rstring = rstring.replace("\r", "\n")
brfox
  • 323
  • 3
  • 14