1

How can I copy Bitbucket repo to CodeCommit, including all branches, but excluding all history on all branches?

Reason for wanting to do this: Our config server reads from CodeCommit, but the source of truth is in Bitbucket. (The reason we do that is a long story.) We have a script that currently (and successfully) "copies" the Bitbucket repo to CodeCommit. We are investigating if we can improve our config server's response times by syncing to CodeCommit without all the history, so when ConfigServer does a clone or fetch, it doesn't take as long (currently 15-25 seconds response times are common).

We hoped we could simply add --depth 1 to our git clone --mirror command. However, when pushing the mirrored repo to CodeCommit, we get an error. To see if it was just a CodeCommit issue, I tried do something similar to read from bitbucket and push back to a new bitbucket repo. I got a different, but similar error.

Here are the core steps. This works just fine without the --depth 1:

git clone --mirror --depth 1 ssh://<bitbucket> RepoFolder
cd RepoFolder
git remote add codecommit ssh://<codecommit>
git remote update origin
aws codecommit create-repository --repository-name pvvts_configs-RepoFolder --repository-description 'Synced'
git push --mirror codecommit

The error occurs on the last command (the git push). The error looks like:

remote: Unknown commit 2280a0e659a1232a402041c27d94d81fd5142edberror: unpack failed: Unknown commit 2280a0e659a1232a402041c27d94d81fd5142edb

Followed by an error such as the following for each branch:

[remote rejected] dev -> dev (unpacker error)

If I try to do something similar except pushing back to bitbucket, I get the following error. (I skip the aws create-repository step, and instead manually create the new repo first in bitbucket.)

error: Could not read 6b0189279671f3a8c7b190dbc8d7ec91c0bb808a To ssh://git@bitbucket.pearson.com/pvvts_configs/vtscoreconfigurations_for_testing_only_marnee_1g_copy.git ! [remote rejected] dev -> dev (shallow update not allowed)

I had read that you could now push from a shallow sync with Git 1.9 or later. I have git version 2.7.4. (See this answer to a question here: https://stackoverflow.com/a/21217267/1599362.) However, perhaps the limitation on pushing from shallow sync was removed from a normal clone, but not from a mirror.

Is there a way to do this shallow sync? Is there some other way to accomplish my purpose of reducing the size of the copied repo in CodeCommit? (Note that config server itself does have an option to tell it to do a shallow clone.)

(I realize even if I can successfully do this "shallow sync", that config server might not play nicely with it, and it might not give the performance improvement that we want. Testing that would be the next step.)

Marnee
  • 370
  • 1
  • 4
  • 15
  • "with all branches but without history" Branches **are** the history. – Code-Apprentice Jan 24 '20 at 16:33
  • I don't think that simplistic statement is true, though being practically a novice with git, perhaps I'm wrong. But consider a repo with a single branch. On that branch is what I would call the "history" of all commits. (Maybe there is a more appropriate term than history?) You could do a depth 1 clone of that repo so that you don't have the history of all commits, yet still have access to that branch. I therefore don't see how one could say they are one and the same. Probably it's just a terminology confusion on my part (my coming recently from Perforce). – Marnee Jan 27 '20 at 20:21

1 Answers1

1

It doesn't appear that CodeCommit lets you make shallow updates. Most Git hosting providers do not, because it means that the repository isn't complete, which prevents certain operations on the server side (such as pull requests) from working. Shallow requests are also very expensive to serve.

This is a feature that's available in Git, as you notede, but different hosting providers support different features of the protocol due to what makes sense for their infrastructure and the fact that many of them use a service other than core Git to handle some or all of the operations in a push.

You can try to make a shallow clone from CodeCommit onto your config server, and that will still work, even if you have a full mirror in CodeCommit. That technique is commonly used in things like CI systems because it makes operations much faster. If you need to update it with an additional fetch, that may be slower than a normal fetch due to the additional work on the server, even if the original clones was faster, but it may be worth a try anyway.

If your goal is to reduce the size of your repository in CodeCommit, then you're kind of out of luck. Unless you want to rewrite history and push a truncated history to CodeCommit, there isn't a way to avoid pushing the entire history there.

bk2204
  • 64,793
  • 6
  • 84
  • 100
  • I was afraid you were going to say that. Thanks much for the detailed explanation._Note: Spring cloud config server does not expose any option for doing a shallow clone at the time. We'd have to inject our own java Git implementation, which we definitely don't want to support. I've submitted a feature request for them to implement, but if what you say is true, it might not help, since fetch has to do the work to filter out history at the point of request._ – Marnee Jan 27 '20 at 20:18