File changed event network share

Question

What I want to do:

I got two directories. Each one contains about 90.000 xml and bak files.

I need the xml files to sync at both folders when a file changes (of course the newer one should be copied).

The problem is:

Because of the huge amount of files and the fact that one of the directories is a network share I can't just loop though the directory and compare os.path.getmtime(file) values.

Even watchdog and PyQt don't work (tried the solutions from here and here).

The question:

Is there any other way to get a file changed event (on windows systems) which works for those configuration without looping though all those files?

I strongly doubt you can get meaningful filesystem notifications from a network share. — millimoose, Feb 23 '13 at 21:38
That's the problem why I asked here... If there's no way of doing it on the current config i may switch to ftp and use ftplib to get the time + file — chill0r, Feb 23 '13 at 22:23
You could create a service running on both machines that monitor their respective folders; when a file changes, capture thiss event (either use the last example in @Jon-Eric 's link or write a service in C# and use the FIleSystemWatcher component. When you detect a change, write this change into a file or message queue or database or whatever, and let this be polled by the other machine to get the list of files to be re-synched. But you'll also need to build some protocol for how your "sync" works when a file has been changed at the same time on both computers :) — Stephen Byrne, Feb 24 '13 at 13:58
Good idea i may look into that. And because of the "What will be if both changed the same time" - That's impossible because it's userdata and a user can only be at one of the servers (but may later use the other one, that's why i want to sync the files). — chill0r, Feb 24 '13 at 15:33

score 2 · Accepted Answer · answered Mar 08 '13 at 15:05

So I finally found the solution: I changed some of my network share settings and used the FileSystemWatcher

To prevent files getting synced on syncing i use a md5 filehash.

The code i use can be found at pastebin (It's a quick and dirty code and just the parts of it mentioned in the question here).

score 1 · Answer 2 · edited May 23 '17 at 12:03

1

The code in...

https://stackoverflow.com/a/12345282/976427

Seems to work for me when passed a network share.

edited May 23 '17 at 12:03

Community

1
1

answered Feb 28 '13 at 18:47

George

1,036
2
11
23

score 1 · Answer 3 · answered Mar 07 '13 at 06:38

I'm risking giving an answer which is way off here (you didn't specify requirement regarding speed, etc) but... Dropbox would do exactly that for you for free, and would required writing no code at all.

Of course it might not suit your needs if you required real-time syncing, or if you want to avoid "sharing" your files with a third party (although you can encrypt them first).

score 0 · Answer 4 · answered Feb 23 '13 at 21:36

0

Can you use the second option on this page?

http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html

answered Feb 23 '13 at 21:36

Jon-Eric

16,977
9
65
97

I tried it but it's not returning anything for a network share (at least in my configuration) – chill0r Feb 23 '13 at 22:22

score 0 · Answer 5 · answered Mar 07 '13 at 09:51

0

From mentioning watchdog, I assume you are running under Linux. For the local machine inotify can help, but for the network share you are out of luck. Mercurial's inotify extension http://hgbook.red-bean.com/read/adding-functionality-with-extensions.html has the same limitation.

In a similar situation (10K+ files) I have used a cloned mercurial repository with inotify on both the server and the local machine. They automatically commit and notified each other of changes. It had a slight delay (no problem in my case) but as a benifit had a full history of changes and easily resynced after one of the systems had been down.

answered Mar 07 '13 at 09:51

Anthon

69,918
32
186
246

"Is there any other way to get a file changed event (on windows systems) which works for those configuration without looping though all those files?" Windows Server 2008 so it's not working. But i found the solution today on my own (i'm adding it as answer, you get the bounty though because your answer is the closest one to my question) – chill0r Mar 08 '13 at 14:55
Sorry, I did not read close enought past the watchdog reference. I once looked at Tim Golden's stuff but I don't work much on Windows. I will have a look at a setup similar to the above which will include Windows clients to see if that would work. – Anthon Mar 09 '13 at 13:07

File changed event network share

5 Answers5