2

I have several dockerized applications that output data to a common directory and a dockerized python watchdog script that watches that directory and is supposed to process the data further once the program output is complete.

It's essential that a file must be complete before it's processed further, I tried to implement a solution similar to Python watchdog windows wait till copy finishes :

def on_modified(event):
    sizeBefore = -1
    while os.path.getsize(event.src_path) != sizeBefore:
        sizeBefore = os.path.getsize(event.src_path)
        time.sleep(1)
    print(f"[+] File transfer of {event.src_path} completed")

which works ok for single files but breaks when it has to monitor multiple files. Waiting until program output is complete and then copy a file to the watched folder is also not an option as I'm dealing with potentially large files that will take at least a few seconds to copy and then I'm having the same issue.

Any ideas on how I can adjust my on_modified event handler to properly watch until a file is not modified anymore, working for multiple files in a single directory

mvk1980
  • 21
  • 4
  • 2
    Can you get your applications to create *"temporary"* files, ie files with particular pattern, eg `"*.tmp"` so that your watchdog script will know that they are in the process of being created and can be ignored. Once a temporary file is fully created your applications can rename them to their final name and your watchdog can take over. – quamrana Aug 31 '20 at 15:47
  • To add to this: in this case you can have the watchdog react only to `on_renamed` events. – MrBean Bremen Aug 31 '20 at 16:11
  • @MrBeanBremen: Can you supply a link to this event? – quamrana Aug 31 '20 at 16:18
  • Ah sorry, remembered that wrong - it is [on_moved](https://pythonhosted.org/watchdog/api.html#watchdog.events.FileSystemEventHandler.on_moved), not `on_renamed`. – MrBean Bremen Aug 31 '20 at 16:27
  • Thanks for the contributions! I tried to avoid changing the docker applications but looks like thats the way forward – mvk1980 Aug 31 '20 at 17:39

0 Answers0