Alternative way to do an initial push of a large repo

Question

I've got a largish Rails 3.1 app in development and production which I've only just set up a staging environment for on Heroku. Because my git repo is quite large, I'm getting time-out errors at around 33% every time I try to push.

Is there an alternative to doing git push staging master for this initial giant push?

The error message is

EmBP-2:Appname Emma$ git push staging master
Counting objects: 17421, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6363/6363), done.
Connection to 10.10.18.33 closed by remote host.46 KiB/s    
error: pack-objects died of signal 13
error: failed to push some refs to 'git@heroku.com:appname-staging.git'

/////////////////// SOLUTION / EDIT, many months later...

There's a sneaky way to solve this, nowadays, using Heroku's (experimental) Pipeline feature, if you already have an environment set up to which you've pushed the code. From the Heroku docs:

"For example, you can push code to staging, have it built into a slug and later promote the staging slug to production."

Takes about 5 seconds for Heroku to push the existing slug from one app to another!

Hey, could you add the new found solution as an answer? I can't implemented yet. Thanks! — Chango, Feb 28 '14 at 14:40
You can find the simple documentation on how to do this here: https://devcenter.heroku.com/articles/labs-pipelines - it worked for me where all the other answers didn't — jfdimark, Mar 18 '14 at 10:02

score 8 · Answer 1 · answered Aug 05 '13 at 07:13

I was trying to push some changes with videos and got:

fatal: The remote end hung up unexpectedly
error: pack-objects died of signal 13
error: failed to push some refs to 'git@github.xxxx/XXX.git'

The solution for me was:

git repack
git push

Hope this will help

score 6 · Accepted Answer · answered Dec 29 '11 at 20:35

6

The alternative is to break apart your giant commit into many small ones. Tag or branch before you do this. Each will have a number of files that constitutes a reasonable push. Make a temp branch to point to the tip. Now reset master to the first of those smaller commits. Push. Set master to the next commit. Push. Repeat this until done.

Now restore master to where it was originally. You already transferred the objects. Pushing this large commit should not resend all the objects that already exist at the remote.

answered Dec 29 '11 at 20:35

Adam Dymitruk

124,556
26
146
141

1

That sounds extremely boring. Ta, though. – snowangel Dec 29 '11 at 20:39
yup. Very yucky but you asked for an alternative ;) – Adam Dymitruk Dec 29 '11 at 20:44
1

I had this error with a `git://` remote repo. I changed to `git+ssh://` and it worked fine. YMMV… – Jean-Philippe Pellet Mar 07 '13 at 13:38
2

You can avoid changing master by checkout out the commits in question, and creating branches from them. You can then push this branch to master on your remote with `git push origin step1:master`. – Luca Spiller Jan 08 '14 at 18:18

javabrett · Answer 3 · 2015-08-10T01:00:52.787

Per Adam's answer, you can break-up your big push containing many commits (and their blobs) into multiple smaller pushes each containing a subset of the required commits for your branch's history.

I have previously done this by breaking the full set of commits in blocks, using temporary branches or tags, but my later attempts use the inline script below. I don't have to move HEAD or modify the index or working-copy.

You will first need to deside how aggressive you will be in terms of the number of commits for each push - a larger number of commits-per-push might be slightly faster, but runs a higher risk of hitting the same problem. Also since your commits will have varying sizes, some periods in the history may contain commits with a higher than average size, so some blocks may succeed whereas others require further splitting. You may also find that you have a single commit that cannot be pushed, and this requires a different solution.

To push to a new remote branch master, run

git log --reverse --oneline | sed -n '0~100p' | awk '{print "git push staging "$1":refs/heads/master"}' | while read i; do eval $i; done

Each push will push the next 100 commits, and only needs to push new objects (or their deltas) found in that batch of commits. Finally, you need to push the current HEAD to push the remainder of the commits, and create the final remote branch:

git push staging HEAD:master

score 0 · Answer 4 · answered Dec 29 '11 at 20:29

0

No, the only way to get content onto Heroku is via a git push.

Out of curiousity how big is your project folder?

answered Dec 29 '11 at 20:29

John Beynon

37,398
8
88
97

The slug size when I deploy to my production env is 72mb. On my local machine, it's 560mb but that includes the sqlite3 db and a bunch of other stuff in gitignore and slugignore. Thanks for the response - at least I know there's no easy alternative. – snowangel Dec 29 '11 at 20:38
do you have lots of static assets in there? Perhaps hosting them on S3 would be better? – John Beynon Dec 29 '11 at 21:08

Alternative way to do an initial push of a large repo

4 Answers4

Linked