11

I've a repo on https://bitbucket.org/

A few days ago by a mistake big number of image files were pushed in the repo. then files were deleted via another push. After that repo worked OK, but today when I try to pull from the repo:

$ git pull
Password for 'https://repo@bitbucket.org': 
warning: no common commits
remote: Counting objects: 4635, done.
remote: Compressing objects: 100% (1710/1710), done.
fatal: Out of memory, malloc failed (tried to allocate 4266852665 bytes)
fatal: index-pack failed

I've tried:

  1. git config --global pack.windowMemory 1024m
  2. $ git count-objects -v count: 9 size: 48 in-pack: 4504 packs: 1 size-pack: 106822 prune-packable: 0 garbage: 0

No luck there, not sure what actions should i take next...
The size of the repo should be around 10-20m of code. what actions should i take next?

UPDATE 1
i executed these commands:

$ git filter-branch --index-filter 'git rm --cached --ignore-unmatch public/images/*' HEAD
Rewrite a1c9fb8324a2d261aa745fc176ce2846d7a2bfd7 (288/288)
WARNING: Ref 'refs/heads/master' is unchanged

and

$ git push --force --all
Counting objects: 4513, done.
Compressing objects: 100% (1614/1614), done.
Writing objects: 100% (4513/4513), 104.20 MiB | 451 KiB/s, done.
Total 4513 (delta 2678), reused 4500 (delta 2671)
remote: bb/acl: ayermolenko is allowed. accepted payload.
To https://repo@bitbucket.org/repo.git
 + 203e824...ed003ce demo -> demo (forced update)
 + d59fd1b...a1c9fb8 master -> master (forced update)

Pull then works ok:

$ git pull
Already up-to-date.

But when I try to clone repo I get

~/www/clone$ git clone git@bitbucket.org:repo.git
Cloning into 'clone'...
remote: Counting objects: 5319, done.
remote: Compressing objects: 100% (1971/1971), done.
fatal: Out of memory, malloc failed (tried to allocate 4266852665 bytes)
fatal: index-pack failed

UPDATE 2
Sadly enough I didn't find all of the large files. Some are still left. So I asked support to kill all the logs of the repo

UPDATE 3
In the end I had to kill old & create a new repo.

Muhammad Dyas Yaskur
  • 6,914
  • 10
  • 48
  • 73
Elmor
  • 4,775
  • 6
  • 38
  • 70
  • 1
    Darn. They (BitBucket) didn't cleaned properly your repo, then. Your solution is a good one though (more practical than mine below), even if you have to change your remote repo address. – VonC Jan 02 '13 at 13:27
  • I got tired of looking for a bad tooth ;) – Elmor Jan 02 '13 at 16:20

4 Answers4

10

In my case, it was something as simple as trying to pull a big repo in a 1 GB RAM box without swap.

I followed this tutorial https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04 to create some swap space on the server and worked.

Their "faster" way:

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

You can make these changes permanent by adding to /etc/fstab:

/swapfile   none    swap    sw    0   0

They recommend adding to /etc/sysctl.conf:

vm.swappiness=10
vm.vfs_cache_pressure = 50
Muhammad Dyas Yaskur
  • 6,914
  • 10
  • 48
  • 73
steinkel
  • 1,156
  • 9
  • 15
4

If you are the only one using this repo, you can follow the git filter-branch option described in "How to purge a huge file from commits history in Git?"

The simpler option is cloning the repo to an old commit, and force push it, as described in "git-filter-branch to delete large file".

Either one would force any collaborator to reset his/her own local repo to the new state you are publishing. Again, if you are the only collaborator, it isn't an issue.

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • collaborators aren't problem. now i've got cloning error. see update 1 – Elmor Dec 26 '12 at 12:45
  • 1
    @Elmor then you need to contact BitBucket in order for them to run a '`git gc`' on your repo (as mentioned in http://stackoverflow.com/a/11404097/6309). They should run it anyway (https://confluence.atlassian.com/pages/viewpage.action?pageId=287998264), but I am not sure if the `git push --force` did trigger said `git gc`. – VonC Dec 26 '12 at 13:22
  • 2
    @Elmor Plus, BitBucket might not clean recent deleted history, so you still might have to contact BitBucket in order to make some more thorough cleaning: `git gc --prune=today --aggressive` or even more, as described in http://stackoverflow.com/a/1908476/6309 – VonC Dec 26 '12 at 13:25
  • Thanks, wrote to the support of bitbucket. waiting for reaction ;) – Elmor Dec 26 '12 at 14:00
2

Even if the big image files have been deleted after having being pushed, they do stay in the git history.

I would suggest to forcibly remove them from the git history (I think that is possible, but it involves a delicate procedure that I don't know).

Alternatively, pull the repository before the mistakenly added files, patch the repository to make the relevant small patches, clone that, and use that (perhaps with a dump/restore) as your master git.

I don't know well the details, but I did read it could be possible

Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
0

I recently encountered this issue with one of my repositories. Similar error, hinting at a large file hidden in the repo somewhere. 

Cloning into 'test_framework'...
remote: Counting objects: 11889, done.
remote: Compressing objects: 100% (5991/5991), done.
Receiving objects:  66% (7847/11889), 3.22 MiB | 43 KiB/sremote: fatal: Out of memory, malloc failed     (tried to allocate 893191377 bytes)
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOFs:  66% (7933/11889), 3.24 MiB
fatal: index-pack failed

To get around this I temporarily created a large swap drive (in excess of the 893191377 bytes the server was asking for) following Method 2 from here: http://www.thegeekstuff.com/2010/08/how-to-add-swap-space/

This allowed me to successfully clone and then remove the culprit (someone had checked in an sql dumpfile). You can use: 

git filter-branch --tree-filter 'rm -rf dumpfile.sql' HEAD

to remove the file from the git repo.

mbarnettjones
  • 131
  • 5