3

I have about 100 000 files on office server (images, pdf's, etc...) Each day files count grows about 100-500 items, and about 20-50 old files changes.

What is the best way to synchronize Web-server with these files? Can any system like Mercurial, GIT help? (On office server, I'll commit changes, and web-server periodically do updates)?

Second problem is, that on Web-server I have user-generated-content (binary-files) (other files). Each day this users upload about 1000-2000 new files. Old files don't change. And I need to backup these files to local machine. Can any system like Merurial, GIT help in this situation? (On web-server I'll commit these files by cron, and on local machine I'll do updates)

Thanks

UPD.

Office server is Windows Server 2008 R2 Web-server is Debian 5 lenny

Lari13
  • 1,850
  • 10
  • 28
  • 55

2 Answers2

5

The simplest and most reliable mechanism (in my experience) is rsync.

On Windows, however, rsync over ssh is badly broken due to issues with how Cygwin interacts with named pipes. Rsync over its own protocol works (as long as you don't care about encryption), but I've had lots of problems getting rsync to stay up as a Windows service for more than a few days at a time. DeltaCopy is a Windows app that uses the rsync tools behind the scenes; it seems to work very well, though I haven't tried the ssh option.

Marcelo Cantos
  • 181,030
  • 38
  • 327
  • 365
  • Now rsync took about 4 hours to do synchronization between office and web-server. It's very long for me. But system was build in 2000, may be I have to upgrade it :) – Lari13 Feb 11 '11 at 12:09
  • Was that the first time you ran rsync, or every time you run it? – Marcelo Cantos Feb 11 '11 at 12:12
  • Are you using the rsync protocol between the source and sink (either directly, or via ssh)? If you are using rsync via a locally mounted file share, it may not give you much benefit. – Marcelo Cantos Feb 11 '11 at 12:15
  • I don't know, becaus this system was made 10 years ago by third persons. Thats why I ask for some new systems to use for this task. – Lari13 Feb 11 '11 at 12:49
4

A DVCS is not a good solution in this case: it will keep the all history, which you don't always need, and will make any clone a massive operation.

An artifact repository like Nexus is much more adapted if you need some kind of versioning with integrity check associated with your binaries.
Otherwise (no versioning), a simple rsync like Marcelo proposes is enough.

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250