1

I am running PollingObserverVFS on a directory that is on Dell ISILON Storage that has over 2 million files and it takes around 2 hours 30 min to detect changes. Is there a way to improve performance to detect changes faster like parallel processing or any other settings changes.

The code is pooling dell isilon storage to detect folder changes.

import datetime
import time 
import os
from watchdog.observers.polling import PollingObserverVFS
from watchdog.events import FileSystemEventHandler


if __name__ == "__main__": 

 # Set format for displaying path 
my_event_handler = FileSystemEventHandler()

path = "\\\\dd\\ARCHIVE\\"

print(path)
# Initialize logging event handler 
#event_handler = LoggingEventHandler() 
def on_any_event(event):

    print(datetime.datetime.now())

    if event.is_directory: 
        if event.event_type == 'created':
             print("Watchdog received directory create event - % s." % event.src_path) 

        elif event.event_type == 'modified': 
            # Event is modified, you can process it now 
            print("Watchdog received directory modified event - % s." % event.src_path) 

        elif event.event_type == 'moved': 
            # Event is modified, you can process it now 
           print("Watchdog received directory moved event - % s." % event.src_path) 
           print("Moved or Renamed directory path - % s." %event.dest_path )

        elif event.event_type == 'deleted': 
            # Event is modified, you can process it now 
           print("Watchdog received deleted event - % s." % event.src_path)


    elif event.event_type == 'created': 
            # Event is modified, you can process it now 
            print("Watchdog received file created event - % s." % event.src_path) 

    elif event.event_type == 'modified': 
            # Event is modified, you can process it now 
            print("Watchdog received modified file event - % s." % event.src_path) 

    elif event.event_type == 'moved': 
            # Event is modified, you can process it now 
           print("Watchdog received moved file event - % s." % event.src_path) 
           print("Moved or Renamed path - % s." %event.dest_path )

    elif event.event_type == 'deleted': 
            # Event is modified, you can process it now 
           print("Watchdog received deleted file event - % s." % event.src_path)        


my_event_handler.on_any_event = on_any_event

observer = PollingObserverVFS(stat=os.stat, listdir=os.listdir,polling_interval=60)
observer.schedule(my_event_handler, path, recursive=True) 

# Start the observer 
observer.start() 
try: 
    while True: 
        # Set the thread sleep time 
        time.sleep(60) 
except KeyboardInterrupt: 
    observer.stop() 
observer.join() 
Vaibav
  • 77
  • 1
  • 7
  • Did you try [inotify](https://pypi.org/project/inotify/) ? – rdas Apr 16 '20 at 04:13
  • Does this answer your question? [inotify with NFS](https://stackoverflow.com/questions/4231243/inotify-with-nfs) – varsh Apr 16 '20 at 04:13
  • we tried watchdog but its not consistent, it does not detect events all the time. inotify with NFS solution is an old solution or suggestion, i was looking if there are any new solution available – Vaibav Apr 16 '20 at 12:18
  • question is revised by the user to be more focused. – Zain Apr 19 '20 at 20:05
  • @Vaibav Did you find a new solution? I've found watchdog to be useful but now have reached the limit with hundreds of thousands of directories being monitored. – skytaker Mar 02 '22 at 16:54

0 Answers0