3

I have a file that I open before a loop starts, and I'm writing to that file almost at each iteration of the loop. Then I close the file once the loop has finished. So e.g. something like:

testfile = open('datagathered','w')
for i in range(n):
    ...
    testfile.write(line)

testfile.close()

  • The issue I'm having is that, in case the program crashes or I want to crash it, what has already been written to testfile will be deleted, and the text file datagathered will be empty. I understand that this happens because I'm closing the file only after the loop, but if I close and open the file after each write (i.e. in the loop) doesn't that lead to an incredible slow-down?

  • If yes, what alternatives do I have for doing the writing, and making sure that in case of a crash the already-written-lines won't get lost, in an efficient way?

  • The linked posts do bring up good suggestions that arguably answer this question, but they don't cover risks and efficiency differences involved. More precisely: Are there any risks involved with playing with the buffersize? e.g. testfile = open('datagathered','w',0) Finally is using with open... still a viable alternative if there are multiple files to write to?

  • Small note: This is asked in the context of a very long run, where the file is being written to for 2-3 days. Thus having a speedy and safe way of doing the writing is definitely valuable here.

  • 2
    Possible duplicate of [Do files get closed during an exception exit?](http://stackoverflow.com/questions/17577137/do-files-get-closed-during-an-exception-exit) – Ilja Everilä Apr 19 '16 at 11:27
  • 2
    Possible duplicate of [Python 2.7 : Write to file instantly](http://stackoverflow.com/questions/18984092/python-2-7-write-to-file-instantly) – Selcuk Apr 19 '16 at 11:28
  • @Selcuk thanks, so if I follow the `testfile.flush()` suggestion given in that post, does that already resolve the matter here? Does it cause a noticeable slowdown? –  Apr 19 '16 at 11:31
  • It should, unless your application crashes during the flush. The slowdown depends on the number of write operations. Why don't you just try it? – Schore Apr 19 '16 at 11:35
  • Perhaps using a database for storing the data might be a good idea. Databases are designed to safely store your data without loosing anything. E.g. sqlite3 is easy to use from Python. – EvertW Apr 19 '16 at 11:51

2 Answers2

6

From the question I understood that you are talking about exceptions may occur at runtime and SIGINT.

You may use 'try-except-finally' block to achieve your goal. It enables you to catch both exceptions and SIGINT signal. Since the finally block will be executed either exception is caught or everything goes well, closing file there is the best choice. Following sample code would solve your problem I guess.

testfile = open('datagathered','w')
try:
     for i in range(n):
         ...
         testfile.write(line)
except KeyboardInterrupt:
    print "Interrupt from keyboard"
except:
    print "Other exception"
finally:
    testfile.close()
msencer
  • 116
  • 3
2

Use a context:

with open('datagathered','w') as f:
    f.write(data)
marienbad
  • 1,461
  • 1
  • 9
  • 19
  • It depends however on the sort of crash the program exhibit. If it is `segfault` this won't help. – robyschek Apr 19 '16 at 11:46
  • Thanks, just as curiosity, any ideas how this contrasts with using `open` and passing 0 as buffer size? –  Apr 19 '16 at 11:46
  • I am not sure, have a look at this answer: http://stackoverflow.com/questions/15991702/what-is-the-difference-between-the-buffering-argument-to-open-and-the-hardcode – marienbad Apr 19 '16 at 12:43