2

To do that, i'm trying to use GIT.

I'm trying to develop a .sh script (under linux) to auto-commit all changes in a git repository and delete all history revision except last one. The repository must preserve only last commit data and clear all previous history (losing all data).

For example, i commit 3 files:

  • file1: 12 MB;
  • file2: 4 MB;
  • file3: 12 KB;

After commit, my .git folder size is approximately 25 MB;

Now i delete file1 and file2.

After next commit my working copy is 12 KB and my .git folder is 25 MB.

I want to flush all data in repository (losing data) to obtain a smallest possible .git folder.

The problem is that I need to do that with ONE SINGLE NON-INTERACTIVE COMMAND LINE, because I need to do a cron script.

Browsing forums for hours I don't found solutions for this question.

The real question is: How to set up a bi-directional fast file sync? The folder to sync is big (20 GB with thousands of files). First sync will be very slow, no problem, but the second one should be faster, no more than 30-40 seconds comparing only changed files...

Thank you for any help.

Rimon Soliman
  • 21
  • 1
  • 5
  • 2
    Hmmm. If you're going to delete history like this, why use git *at all*? If you insist, have a look at the `git filter-branch` command – brice Apr 03 '12 at 08:46
  • I'm using git because I'm trying to develop a completly automated file-syng git-based... :-) In file-sync i don't need history. I alredy tryed git filter-branch but not working... – Rimon Soliman Apr 03 '12 at 09:00
  • file-sync? => why not `rsync`? – ArjunShankar Apr 03 '12 at 09:11
  • right question, but sync must be bidirectional and automatically executed every minute. rsync (or other sync system) do not use database or any file index and every run will compare ALL local and remote files. i have many GB working copy in thousands files and rsync will be very slow... (i think). if you know how to set-up a bidirectional fast sync with rsync please teach me :-) – Rimon Soliman Apr 03 '12 at 09:23
  • You left these details out in the question. And IMO, hacking up git to do this might not be the best way. I think your question should be "How to set up a bi-directional fast sync?" Hopefully, others know. – ArjunShankar Apr 03 '12 at 09:32
  • What I meant is: Consider 'edit'ing your question to reflect your comment. – ArjunShankar Apr 03 '12 at 09:33
  • Ok, i edited my question title. Thanks for suggest. – Rimon Soliman Apr 03 '12 at 09:44

3 Answers3

1

How to set up a bi-directional fast file sync would be osync. Osync is rsync based bi-directionnal file sync and will use rsync update algorithm to spare some bandwidth for propagating only changes. Check it out at http://www.netpower.fr/osync

Orsiris de Jong
  • 2,819
  • 1
  • 26
  • 48
1

This Python script I coded may help. Give it a try!

https://github.com/dooblem/bsync

Marc MAURICE
  • 1,571
  • 1
  • 11
  • 9
1

Get it working fast

Use AeroFS or unison.

Custom solution

Use inotify for linux, or its equivalent on other platforms to get notified of changes to your file system, then use librsync, libssh or similar to transfer the files across.

Depending on your usage, you might actually get away with running rsync in a cron job. The rsync algorithm is pretty darn smart, and will transmit very little unnecessary crud when used. If you set it up properly, it will only look at last modified time.

I would definitely recommend using either aerofs or unison before trying to put together a ad-hoc solution.

Community
  • 1
  • 1
brice
  • 24,329
  • 7
  • 79
  • 95