7

We have a Git repo with 7 contributing developers with over 2.5 years of history and about 10,000 commits. We use Assembla to push and pull from. When we add new developers cloning the repo to their dev computers takes almost an hour.

I'm not sure if this is the proper terminology, but our goal is to "shrink" the repo by "cutting/snipping" off the first 1.5 years worth of commits and keeping only the latest year of history. We want to keep a "backup" copy of the entire repo, whether as a separate repo or maybe a branch? We'd like to repeat this in the future an possible merge the initial split with the new split when needed, but I'm not sure if this is possible. If there is a way to have all the history on a separate branch and just keep the master branch with only the history for the last year, that would be great, but let me know of possible pros and cons.

I don't know of all the possibilities/options we have which is why I am here. I read something about patches but I'm not sure if that truly is what I need or if there is something better/easier. What are you guys doing to take care of an issue like this, including pros and cons? Keep in mind, I still need every developer to continue to push and pull, preferably staying on the master branch.

Thanks in advance!

RoLYroLLs
  • 3,113
  • 4
  • 38
  • 57
  • How big is your repo (disk size)? Git should be able to handle this amount of commits. You may also look for excess load on the server or network problems (as a comparison, the linux kernel is about 800MB large, has 400K commits and takes here about 1/2 hour to get cloned from git.kernel.org). – Rudi Nov 21 '13 at 22:20
  • Rudi, the repo is about 30GB with over 10K commits (the .git/objects/ folder is 12GB), nothing compared to the linux kernel. However, beyond any possible network issues, I'm not just trying to fix then and move on, I'm also wanting to learn different methodologies available regarding this. Thanks! – RoLYroLLs Dec 20 '13 at 23:32
  • possible duplicate of [Git remove history commit](http://stackoverflow.com/questions/13091928/git-remove-history-commit) – sschuberth Aug 10 '15 at 19:06

2 Answers2

7

The best step-by-step instructions can be found on the Git SCM historical blog post "Replace Kicker" or in the Git book's chapter on "Replace".

The short summary is this:

  • Create a new branch that is at the point where you want to cut, say git branch history hash.
  • Push the history to a new repository.
  • Create a new base using git commit-tree.
  • Rebase your post-history commits onto your new base.
  • Push your new truncated master branch up to the server.
  • People can then use git replace to re-connect the history together.

The original post explains it much better with pictures.

When dealing with complex histories involving merges, this may not work well, depending on how well git rebase --onto works with --preserve-merges. You should obviously test well before proceeding.

dseifert
  • 1,318
  • 9
  • 22
Emil Sit
  • 22,894
  • 7
  • 53
  • 75
  • Thanks! I'm actually liking this process. I was just testing this method out. DO you know if there is a way to keep this method half way? IE: `master` branch have the truncated version, and the `history` branch keep the entire history up until i split them? I prefer not to have to push to a separate repo. – RoLYroLLs Dec 20 '13 at 16:30
  • To add to my comment above, I just read that a `git pull {remote}` pulls all branches but doesn't create a local branch for any/all branches on the remote repo. if I manage to keep the history in a branch and all new commits on another branch, will new developers end up pulling ALL data from each branch (which I guess was the reason for `push`ing to a new repo in your answer), or does it only pull the branch names and not the data? – RoLYroLLs Dec 20 '13 at 16:58
  • I did some tests on another repo and got the answer to my question about how much data does `git pull` get. Indeed, it DOES get all branches and their tracked changes, even if you do not have a local branch tracking the changes to a specific branch. So my idea of keeping the history on a separate branch will not be feasible. I'll have to follow the Replace Kicker post you posted! Thanks! – RoLYroLLs Dec 21 '13 at 00:21
0

Disclaimer: You should do this with a test repo first, since the commands listed here can destroy your data.

You can edit (i.e. let a script edit) the history via the fast-export mechanism. The first step is to run git fast-export --signed-tags=strip --no-data --full-tree --export-marks=export.marks branch1 branch2 [...] branchN > commits.fi. Now you have a fast-import stream of all your branches. In this stream, you can drop a commit by deleting the lines from commit refs/... to the trailing newline. You need also to remove the from :<mark> line (and also merge :<mark> lines if your new history starts with a merge) from your then new "first" commits.

To aid this process, you should look into the revision graph, and take revisions where no merges crosses their history. In the following graph, A, D and F are good candidates to start with. B or C do have the problem, that the successor D does also depend on G and A. 

A ---- B ---- C ---- D --- E ---- F
  \                /  \     \   /
   \--- G ------- H    \--I--J-/

With the export.marks file you can translate the commit ID to mark numers in the stream.

You need to give your branches new names, since git fast-import won't accept the new history, because the new branches don't contain commits from the existing ones.

After you created the new history, you need to import it with git fastimport < manipulated-history.fi into your existing repo.

Now it is time to check if the import was correct. For this you need to clone the repo into a temporary one, and in this temporary repo you create for each newly created commit a graft so that it has the same parent revisions like before. Afterwards you run git filter-branch newBranch1 newBranch2 [...] newBranchN. The import was correct if each newBranch now stays at the same commit as the corresponding branch.

When everything worked so far, you can create a new repo and pull the newBranch-branches from the first working clone int it, and make it the new working repo. Also note that you should not leave any repos with grafts anywhere, since grafts can cause harm.

Rudi
  • 19,366
  • 3
  • 55
  • 77
  • @Rubi thanks for this. not sure how well this solution will work due to the graphs issue you mentioned. – RoLYroLLs Dec 20 '13 at 16:34