0

I am trying to use a file (csv, json, txt, haven't decided format) that I can drop a few lines of data in. A python script will be on cron to run every 5 minutes and check the file if there is anything new and if there is, process it and delete each line as it is processed.

I am trying to prevent a situation where I open the file, make some changes and save it while the process came by grabbed the data and emptied the file but my save writes it back in.

I thought the only way to make this safe is to have it process a folder and just look for new files, all changes would be dropped in a new file. So there is never the risk of this happening.

Is there a better way, or is this the best approach?

Michael
  • 525
  • 1
  • 8
  • 20
  • 1
    Not sure if this is what you may be looking for: https://stackoverflow.com/questions/30407352/how-to-prevent-a-race-condition-when-multiple-processes-attempt-to-write-to-and – SajidSalim Sep 14 '17 at 06:43
  • I second the above suggestion. Filesystem level locks are good enough for this purpose. Also you could try instead running the python script as a daemon handling inotify events if your filesystems supports it. See [suggestions for inotify and crossplatform solutions in this post](https://stackoverflow.com/q/12582720/1328439) – Dima Chubarov Sep 15 '17 at 10:46

2 Answers2

1

Check this answer to see if the file is already open and if it is, just wait 5 more minutes until, or alternatively sleep internally and try again every 10 seconds until it works, but no longer than 4 minutes, for example:

for i in range(attempts):
    if not fileInUse():
        processFile()
    else:
        time.sleep(10)
Carl
  • 2,057
  • 1
  • 15
  • 20
0

You can use the below steps:

  1. Python script which runs in cron will check if the file is opened by any other process. In Linux, it can be done using lsof.
  2. If file is open, when cron runs, then it will not process the file data.
  3. Same logic can add for the script which will add data to the file, if the file is used by some other scripts.
tripleee
  • 175,061
  • 34
  • 275
  • 318
Dinesh Pundkar
  • 4,160
  • 1
  • 23
  • 37