20

I noticed in the docs they always open a CSV file with ‘wb’. Why the ‘b’? I know b stands for binary mode, but when do you use binary mode (I’d guess CSV file is not binary). If relevant I’m writing to the CSV from results from query by arcpy.da.SearchCursor()

EDIT: just noticed according to this answer wb+ is used for writing a binary file. What does including the + do?

Community
  • 1
  • 1
Celeritas
  • 14,489
  • 36
  • 113
  • 194
  • 1
    if there are any non-ascii characters you should use binary mode ... opening in binary mode will never alter or harm your data afaik , however sometimes opening in ascii mode will delete or change characters ... as such I just always open in binary mode (even with ascii data) – Joran Beasley Jul 17 '15 at 19:25
  • 1
    @JoranBeasley: there is not such thing as "ascii mode". Use the term "text mode" instead (the character encoding of a text file can be anything). – jfs Jul 18 '15 at 14:17
  • 1
    read answers to [the question you've linked](http://stackoverflow.com/questions/16208206/confused-by-python-file-mode-w) -- `+` means that you want both to write *and* to read from the file. – jfs Jul 18 '15 at 14:20

6 Answers6

14

Use 'b' mode, to read/write binary data as is without any transformations such as converting newlines to/from platform-specific values or decoding/encoding text using a character encoding.

csv module is special. csv data is text and therefore the text mode would be expected but csv module uses '\r\n' by default to terminate rows on all platforms and it always recognizes both '\r' and '\n' as newlines. If you open the corresponding file in the text mode (with universal newlines) then you will get '\r\r\n' (corrupted newlines) on Windows (os.linesep == '\r\n' there). That is why Python 2 docs say that you must use the binary mode. In Python 3, the text mode is used but you should pass newline='' to disable universal newlines mode. You would also want to disable universal newlines if you want to preserve possible newline characters (such as '\r') embedded in fields.

jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • To be picky, passing `newline=''` doesn't disable universal newlines mode. From the [docs](http://example.com): "If it is '', universal newlines mode is enabled, but line endings are returned to the caller untranslated." – user_ Apr 02 '19 at 22:50
6

File open default is to use text mode, which may convert '\n' characters to a platform-specific representation on writing and back on reading.

In windows this will modify the line breaks from '\n' to '\r\n' which will create problem opening the CSV file in other applications/platforms.

Thus, when opening a binary file, you should append 'b' to the mode value to open the file in binary mode, which will improve portability. On systems that don’t have this distinction, adding the 'b' has no effect.

Note: 'w+' truncates the file.

Modes 'r+', 'w+' and 'a+' open the file for updating (reading and writing).

As detailed here: https://docs.python.org/2/library/functions.html#open

CodePick
  • 61
  • 1
  • What is "truncating" a file, does it mean replace? – Celeritas Jul 17 '15 at 20:45
  • It is the opposite. Text mode increases portability of your code (you can read text written on other platforms, files written locally are understood by platform-specific programs such as `notepad.exe`). [`csv` is a special case where binary mode is used for text data](http://stackoverflow.com/a/31492038/4279) – jfs Jul 18 '15 at 14:36
  • Truncate means the file will be created if it doesn't exist and file is overwritten if it exists. So yes in simple term it means replace. – CodePick Jul 19 '15 at 07:29
  • I agree, "Text mode increases portability and csv is just a special case where binary mode is used for text data". – CodePick Jul 19 '15 at 07:41
1

By using t on non-Posix environments (like MSDOS and MS Windows), the \r\n sequence is transformed into \n on input (and the opposite on output). b (binary mode) performs no such translation.

Presumably the CSV library deals with carriage returns (probably by ignoring them whenever it encounters them).

Edit: just noticed a changed question.

Since .CSV files aren't really intended for human readers, the library can output them with \n (linefeed (LF) aka newline) separators only. They only real downside would be a MSWindows user opening the file with Notepad: it will display poorly. The CSV library could also output files with \r\n (CR LF) since most programs defend against MSDOS text file conventions.

Either way, the library can write through b (binary) mode just fine. It is possible that if opened in t (text) mode, the line separators would have something slightly odd like \r\n\n. Probably most CSV file parsers ignore the CR, and recognize LF LF as ending a line and following it with an empty (blank) line, which it also ignores.

The + is explained in the man page:

w+      Open for reading and writing. The file is created if it does not exist, otherwise it is truncated. The stream is positioned at the beginning of the file.

The difference is that w+ allows reading and writing whereas w only allows writing.

wallyk
  • 56,922
  • 16
  • 83
  • 148
  • 1
    probably by using them as row delimiters (`\n`) ... good answer though – Joran Beasley Jul 17 '15 at 19:30
  • whether or not `'\r\n'` is translated in text mode is platform-dependent (`os.linesep`). It does on Windows. It doesn't on OS X. You don't need to use `'t'` mode explicitly (it is passed to stdio functions on Python 2 and therefore it may enable platform-dependent behavior such as recognizing Ctrl+Z as an EOF character on input). [`csv` module always recognized both '\r' and '\n' on input](http://stackoverflow.com/a/31492038/4279) – jfs Jul 18 '15 at 14:28
0

Ive never recieved a good explanation on why I shouldnt just open ascii files in binary mode.

I have never seen opening a file in binary mode to corrupt the data.

I have seen opening the file in ascii mode, alter or harm the data being retrieved, ergo I and I assume most seasoned python programers in general will open files in binary mode unless we have some sort of guarantee that there is not and never will be binary characters in the file.

Joran Beasley
  • 110,522
  • 12
  • 160
  • 179
  • to understand why, write `open('binary.txt', 'wb').write(b"1\n2\n3\n")` and `open('text.txt', 'w').write("1\n2\n3\n")` and try to open both files in the `notepad.exe` – jfs Jul 18 '15 at 14:39
  • because notepad.exe is too dumb to recognise \n as a newline when it is not preceded by \r is not really a good reason for me ... if I write "\n" I expect the file to only contain \n... – Joran Beasley Jul 18 '15 at 21:55
0

Since opening a file in text mode relegates handling of newlines differently based on the operating system to core code, the CVS routine authors must have determined they wanted more control - that they would prefer to handle newlines themselves. This may have allowed them to resolve bugs from inconsistencies encountered processing files under one OS that were created on another OS -- where "text read" altered things problematically in some unique cases. It also may be no bugs were found but they wanted to avoid future possibility. Or, it may also be that since they had to deal with newline considerations anyway, bypassing text processing might be faster.

Logically, since one can't control the OS source of a file being read, then using binary might be the better way to go in general. However, writing a text file one might do well to leave it up to the core routines to handle newlines for the current OS by using text mode.

The "+" is discussed at Confused by python file mode "w+"

Community
  • 1
  • 1
Cris Mooney
  • 185
  • 10
-1

For the Python csv module in particular, the answer is simple: it's required by the documentation.

If csvfile is a file object, it must be opened with the ‘b’ flag on platforms where that makes a difference.

Source: https://docs.python.org/2.7/library/csv.html#csv.reader

Daniel Pryden
  • 59,486
  • 16
  • 97
  • 135