15

We're just moving to git at my work. We have a reasonably large project with some fairly large resources under version control (~500MB).

Today we tried to push a branch to a remote server and were surprised that git seemed to be trying to upload the entire project. I would have expected git to only send the deltas for the 4-5 text files that had changes (like it does for a normal push to master). Is this not how remote branches work? We used git push origin some_branch_name, is there a better command to use in this case? Should we not be storing large resources in git? If not, how do people usually handle this scenario? Is there a better way for one developer to share in-progress work with another developer w/o committing the changes to the master branch? As it stands, we're looking at around 15 min to push a remote branch, which is really not workable. What are we doing wrong?

herbrandson
  • 2,357
  • 2
  • 30
  • 44
  • Did you try `git push origin HEAD:some_branch_name`? – devang Mar 22 '13 at 04:20
  • 1
    Same result. I'm testing on a branch that removes one file. After running for a few minutes it reads `Writing objects: 17% (8153/46698), 192.53 MiB | 2.46 MiB/s` – herbrandson Mar 22 '13 at 04:26
  • 1
    How long has it been since you've run `git gc` on the local repository? I'm not positive that's what's going on (I'm not sure why that would lead to writing more data to the network, for example), but I've had it substantially speed up a huge variety of git operations. – rra Mar 22 '13 at 04:38
  • 1
    @rra 'git gc' didn't help – herbrandson Mar 22 '13 at 18:03

2 Answers2

2

Can you update your post with a few things?

To get a better idea of how your project looks, please post around the top 10+ entries of the following:

git log --decorate=short --oneline --graph --all

If the large resources are binaries, then no they should not be stored in git. If those binary resources are updated then git then it has to make a complete duplicate of them internally, which the compression algorithm doesn't like, and send those up to the server. As for what to do about them, that depends on the scenario. You'll need to elaborate.

It sounds like you have several developers working on the same remote. Is this correct? If so, no developer should be committing directly to master (imho should never happen anyways). It's possible for each developer to have their own named branch. For example, developer John can create all his branches under john/<branch_name>. This will help keep the workflow clean.

Also, git doesn't work with deltas. It stores the file in its entirety each time it's changed. This may seem inefficient, but the compression used keeps size down to a minimum. And it helps checkouts and scanning log history much faster. Read the first section of Git Basics for a visualization.

Trevor Norris
  • 20,499
  • 4
  • 26
  • 28
  • 5
    this doesn't explain why git is pushing the whole repo (or feels like it, anyway), when only a few files have changed. the fact that there are lots of binary files shouldn't make a push (that doesn't change any) slower – grahamrhay Mar 18 '15 at 09:53
  • I think it may be related to this https://stackoverflow.com/questions/29118876/git-push-is-very-slow-for-a-branch perhaps? Doing a git fetch sets it up for quick transport. Or something like that. – Phil Martin Feb 11 '16 at 12:22
1

Another factor can explain the git push poor performance:

"git push" used to check ambiguities between object-names and refnames while processing the list of refs' old and new values, which was unnecessary (as it knew that it is feeding raw object names).

See commit a4544b3 (06 Nov 2018) by Derrick Stolee (derrickstolee).
Helped-by: Jeff King (peff).
(Merged by Junio C Hamano -- gitster -- in commit 1373999, 19 Nov 2018)

pack-objects: ignore ambiguous object warnings

A git push process runs several processes during its run, but one includes git send-pack which calls git pack-objects and passes the known have/wants into stdin using object ids.

However, the default setting for core.warnAmbiguousRefs requires git pack-objects to check for ref names matching the ref_rev_parse_rules array in refs.c.
This means that every object is triggering at least six "file exists?" queries.

When there are a lot of refs, this can add up significantly!
I observed a simple push spending three seconds checking these paths.

This has been fixed with Git 2.20 (Q4 2018).

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250