4

I'm using a Queue and threads to write data to a CSV. I'm running into a problem where one worker will be writing to the CSV, and then another will write to the CSV, causing issues in how the data is laid out.

Is it possible to lock the CSV when one worker opens it, so that no other worker can open it until they have closed it again?

Thanks!

chris
  • 1,869
  • 4
  • 29
  • 52
  • Have you looked at calling `chattr` from the system (http://stackoverflow.com/questions/11455468/how-to-create-a-file-lock-on-any-txt-php-css-or-other-filetypes-on-linux-ubunt) – Steve Mar 10 '16 at 18:51
  • hmm but can you call that from within the python script? also would this work for mac? – chris Mar 10 '16 at 18:53
  • You could also have a look at these options: http://stackoverflow.com/questions/489861/locking-a-file-in-python – Steve Mar 10 '16 at 18:54
  • `chattr` should work on a mac and linux but not windows. – Steve Mar 10 '16 at 18:54

1 Answers1

0

You have two basic options. One is to use the locking provided by the threading module within your application. A simple Lock object will likely suffice. This can be used in general to protect any resource against concurrent access by multiple threads. However, it won't do any good if your access spans multiple instances of your process.

Your other option is to use filesystem locking with the fcntl.lockf() function. This allows multiple processes to protect the file against concurrent access, but note that this is still advisory locking - it doesn't do anything to protect you against another process writing to the file which isn't using your locking strategy.

The advantage of the threading locks is that they'll be lighter weight and faster. The advantages of the filesystem locking is that it works between processes and also reduces the need for you to pass state around inside your application (i.e. the lock objects). Unless performance is a paramount concern I'd go with filesystem locking. The portability of the filesystem locking may be limited, however, in which case a standard Lock object may work out better for you.

Whichever one you use, you should ideally make sure you use the with statement to ensure you always release the lock when you're done with it. Note that filesystem locking is also automatically released when you close the file handle, however.

Cartroo
  • 4,233
  • 20
  • 22