1

I have a huge number of git repositories on an on-site linux server that I need to back up daily to an off-site windows server. Because there are so many files, I want to use rsync instead of plain copy to save time and network bandwidth. (I will use rsync after mounting the windows destination drive.) I also want to avoid a solution that tars up all the files or otherwise compresses them into one big file. This is because the offsite-server is further replicated to other offsite servers, and if I uses a big tar file, instead of just having to copy the 1 or 2 small files that changed, the entire big tar file will have to be copied. (*1)

My question is, should I or should I not use the delete flag (--delete) on rsync?
The delete flag on rsync will delete files found on the destination but not on the source. I would ideally like to leave the delete flag off, because if a wanted file accidentally got deleted from the source, it would also get deleted on the destination. By no using the delete flag, we risk having the destination be a superset of all the files we want.

Is this a problem? Would this somehow corrupt the git repositories on the destination? If it makes a difference in your answer, we only allow fast-forward commits to be pushed to our on-site git server. Maybe it's the case that git will never delete files in the .git directory if only fast-forward commits are done?

(*1) Edit 1: Add note about wanting to avoid solutions that compress multiple files into one. Thus git bundle wouldn't be a wanted solution. If rsync isn't the way to go, I'd love to know of any alternative approach you recommend.

user64141
  • 5,141
  • 4
  • 37
  • 34
  • The bup tool that I mention doesn't tar everything in one big file, so it fits your requirements. – VonC Nov 01 '13 at 22:21

1 Answers1

0

The issue isn't so much about the --delete option (which I would use to keep a consistent image of the repos on both sides), and more the risk of corruption when copying so many files.

One solution would be to have a job updating incrementally local (to the server) bundles.
A bundle is like a git repo, but condensed in one file.
There, your rsync would have to copy over only one file per repo, but that wouldn't backup local config and local hooks of each repos.

The other solution is to use an rsync-like solution, able to backup incrementally a huge volume of data: bup (presented in Git Minutes #24).

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250