18
> git gc --aggressive --prune=now
Counting objects: 68752, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (66685/66685), done.
fatal: sha1 file '.git/objects/pack/tmp_pack_cO6T53' write error: No space left on device

sigh, ok

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        19G   15G  3.0G  84% /
udev            485M  4.0K  485M   1% /dev
tmpfs            99M  296K   99M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            494M     0  494M   0% /run/shm
cgroup          494M     0  494M   0% /sys/fs/cgroup

doesn't look that bad

ls -lh .git/objects/pack/
total 580M
-r--r--r-- 1 foouser root  12K Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.idx
-r--r--r-- 1 foouser root 5.1M Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.pack
-r--r--r-- 1 foouser root 5.1K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.idx
-r--r--r-- 1 foouser root 100K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.pack
-r--r--r-- 1 foouser root  11K Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.idx
-r--r--r-- 1 foouser root 2.6M Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.pack
-r--r--r-- 1 foouser root 1.6M Apr  3  2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.idx
-r--r--r-- 1 foouser root 290M Apr  3  2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.pack
-r--r--r-- 1 foouser root  40K Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.idx
-r--r--r-- 1 foouser root 6.1M Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.pack
-r--r--r-- 1 foouser root 1.6M Apr 19  2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.idx
-r--r--r-- 1 foouser root 102M Apr 19  2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.pack
-r--r--r-- 1 foouser root 1.6M Oct  3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.idx
-r--r--r-- 1 foouser root 151M Oct  3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.pack
-r--r--r-- 1 foouser root 4.7K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.idx
-r--r--r-- 1 foouser root 125K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.pack
-r--r--r-- 1 foouser root 6.2K Oct  2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.idx
-r--r--r-- 1 foouser root 4.2M Oct  2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.pack
-r--r--r-- 1 root          root  16M Feb 27 08:19 tmp_pack_cO6T53

So, git gc bails out on a tmp pack that's only 16MB big while my disk appears to have 3GB free. What am I missing? How can I get git gc to work more reliably? I've tried without aggressive option and --prune instead of --prune=now as well, same story.

Update

Doing a df -h during the repack action it shows that it is now using all my disk (100% usage). A little while later the repack action fails and it leaves another 14MB file in the .git/objects/pack/ folder. So, to recap, my packs use a total of 580MB. git repack somehow manages to use up 3GB to repack that. I have ~800MB free in the RAM after it's done btw. - maybe it's using so much working memory that it clogs up the swap? I guess my question comes down to: Are there options to make git repack less resource hungry?

versions: git version 1.7.9.5 on Ubuntu 12.04

Update 2 I've updated git to 2.3. Didn't change anything unfortunately.

> git --version
git version 2.3.0
> git repack -Ad && git prune
Counting objects: 68752, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (36893/36893), done.
fatal: sha1 file '.git/objects/pack/tmp_pack_N9jyVJ' write error: No space left on device

Update 3

Ok, so I just noticed something that is curious: the .git directory actually uses much more disk space than the 508MB previously reported. 

> du -h -d 1 ./.git
8.0K    ./.git/info
40K ./.git/hooks
24M ./.git/modules
28K ./.git/refs
4.0K    ./.git/branches
140K    ./.git/logs
5.0G    ./.git/objects
5.0G    ./.git

Upon further inspection .git/objects/pack actually uses 4.5GB. The differences lies in hidden temp files I didn't notice before: 

ls -lha ./.git/objects/pack/
total 4.5G
drwxr-xr-x   2 foouser root  56K Feb 27 15:40 .
drwxr-xr-x 260 foouser root 4.0K Oct 26 14:24 ..
-r--r--r--   1 foouser root  12K Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.idx
-r--r--r--   1 foouser root 5.1M Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.pack
-r--r--r--   1 foouser root 5.1K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.idx
-r--r--r--   1 foouser root 100K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.pack
-r--r--r--   1 foouser root  11K Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.idx
-r--r--r--   1 foouser root 2.6M Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.pack
-r--r--r--   1 foouser root 1.6M Apr  3  2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.idx
-r--r--r--   1 foouser root 290M Apr  3  2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.pack
-r--r--r--   1 foouser root  40K Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.idx
-r--r--r--   1 foouser root 6.1M Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.pack
-r--r--r--   1 foouser root 1.6M Apr 19  2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.idx
-r--r--r--   1 foouser root 102M Apr 19  2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.pack
-r--r--r--   1 foouser root 1.6M Oct  3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.idx
-r--r--r--   1 foouser root 151M Oct  3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.pack
-r--r--r--   1 foouser root 4.7K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.idx
-r--r--r--   1 foouser root 125K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.pack
-r--r--r--   1 foouser root 6.2K Oct  2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.idx
-r--r--r--   1 foouser root 4.2M Oct  2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.pack
-r--r--r--   1 root          root 1.1K Feb 27 15:37 .tmp-7729-pack-00447364da9dfe647c89bb7797c48c79589a4e44.idx
-r--r--r--   1 root          root  14M Feb 27 15:29 .tmp-7729-pack-00447364da9dfe647c89bb7797c48c79589a4e44.pack
-r--r--r--   1 root          root 1.1K Feb 27 15:32 .tmp-7729-pack-020efaa9c7caf8b792081f89b27361093f00c2db.idx
-r--r--r--   1 root          root  41M Feb 27 15:30 .tmp-7729-pack-020efaa9c7caf8b792081f89b27361093f00c2db.pack
-r--r--r--   1 root          root 1.1K Feb 27 15:37 .tmp-7729-pack-051980133b8f0052b66dce418b4d3899de0d1342.idx
(continuing for a *long* while).

Now I'd like to know: Is it safe to just delete those?

Michel Müller
  • 5,535
  • 3
  • 31
  • 49
  • As seen in http://www.spinics.net/lists/git/msg200279.html, do you have `.git/objects/*/tmp_*` files? – VonC Feb 27 '15 at 07:47
  • As I mentioned in http://stackoverflow.com/a/28563637/6309, would a `git repack -Ad` followed by a `git prune` help? – VonC Feb 27 '15 at 07:48
  • @VonC: Thanks you for your help. Yes, the one you see in my ls-output. there are no other objects-subfolders that contain tmp-files. – Michel Müller Feb 27 '15 at 07:48
  • I'm doing `git repack -Ad` right now, it takes quite a while. I'll let you know. – Michel Müller Feb 27 '15 at 07:54
  • Doing a `df -h` during the repack action it shows that it is now using all my disk (100% usage). A little while later the repack action fails and it leaves another 14MB file in the `.git/objects/pack/` folder. So, to recap, my packs use a total of 580MB. `git repack` somehow manages to use up 3GB to repack that. I have ~800MB free in the RAM after it's done btw. - maybe it's using so much working memory that it clogs up the swap? I guess my question comes down to: Are there options to make `git repack` less resource hungry? – Michel Müller Feb 27 '15 at 08:02
  • The main point of http://stackoverflow.com/a/28563637/6309 was that `git repack` alone isn't enough: `git repack -Ad` + `git prune` should reduce the size of the repo. – VonC Feb 27 '15 at 08:09
  • What version of Git are you using by the way? And on what OS? – VonC Feb 27 '15 at 08:09
  • > What version of Git are you using by the way? And on what OS? -- git version 1.7.9.5 on Ubuntu 12.04 LTS. – Michel Müller Feb 27 '15 at 08:30
  • > git repack alone isn't enough -- well, it still doesn't help if repack already fails because it blows up to 3GB, does it? Or do you mean I should just do those two commands 10 times in a row because it's supposed to get a bit smaller every time, so repack can get further? – Michel Müller Feb 27 '15 at 08:31
  • I missed the fact the git prepack didn't complete. Just in case it changes anything, coud you see if the error persists with a git 2.3.1? (http://stackoverflow.com/a/24847953/6309) – VonC Feb 27 '15 at 08:35
  • Ubuntu packages seem to be at 2.3.0. https://launchpad.net/~git-core/+archive/ubuntu/ppa Does that make a difference, i.e. would you recommend building from source? – Michel Müller Feb 27 '15 at 08:45
  • No 2.3.0 is close enough. – VonC Feb 27 '15 at 08:46
  • Note that on a typical filesystem formatted under Linux, certain space gets reserved for the root user. By default, it's typically 5%, which was sensible back in the day but supposedly not so nowadays with terabyte disks. So, `df` shows you total space available on the filesystem while your non-root account is able to use less that that. Please check to see if that's not really the case. See [this](http://unix.stackexchange.com/q/7950/43156) for more info. – kostix Feb 27 '15 at 09:37
  • @kostix: thanks for the interesting info, but I don't think it's the case here since df -h reports 100% disk usage during the repack action. If it were for the 5% it would only go up to 95%, correct? – Michel Müller Feb 27 '15 at 10:41
  • @MichelMüller, no, `df` always reports the full space available no matter who's calling it. I mean, it just asks the filesystem for its free space. If you then will try to fill that space using a non-root account, you'll hit the limit before all that space will be occupied. If you're on an `extN` filesystem, try using `tune2fs -r 25600 /dev/sdX` to set that root-reserved space to 100MiB and see if that helps. You'll be able to revert it back later. – kostix Feb 27 '15 at 12:16
  • @kostix: isn't that what I wrote? It reports 100%. Which according to you is a reflection of what's going on on the disk, not some OS allotments. If the 5% allotment were what's causing the issue, could it report 100%? It goes from the 84% as shown above right up 100% - this delta, together with the 3GB reported, makes me think that I don't have to search the issue in the OS or the FS. – Michel Müller Feb 27 '15 at 15:01
  • @VonC: See Update2, I just updated to 2.3.0 and it didn't change the outcome. – Michel Müller Feb 27 '15 at 15:02
  • @MichelMüller darn it ;) Any chance you could raise the disk space? 20GB looks like a vdi (virtual disk image used in a VirtualBox setting) – VonC Feb 27 '15 at 15:15
  • @VonC: Yes it's in VMware Fusion. Unfortunately it doesn't automatically grow the partition itself with VMware, so the process is kinda involved (usually I boot into a GParted Live CD to do this). Also, this is on a Laptop with 256 GB SSD that also has a Win8 VM and a few other rather space consuming things, so I don't really have much space to throw at it. – Michel Müller Feb 27 '15 at 15:45
  • @VonC: See my last edit: Now I know what takes up so much space and why the repack blows up - I'm actually dealing with 4.5GB, not just 500MB. What do you think, should I just risk it and delete these hidden tmp files? If things turn sour I should always be able to just clone again, no? – Michel Müller Feb 27 '15 at 15:53
  • @MichelMüller not sure: that was what http://www.spinics.net/lists/git/msg200279.html (the link in my first comment) seemed to do. – VonC Feb 27 '15 at 16:11
  • @MichelMüller my second command referenced a question which used `git count-objects -v -H`: that should also give you an idea of the size of the repo. – VonC Feb 27 '15 at 16:12
  • @VonC: Please note: your link deals with *visible* tmp files, not hidden ones. So this here seems to be a new case. – Michel Müller Feb 27 '15 at 23:16

4 Answers4

8

So here is what I found out so far: I couldn't find any documentation about these hidden '.tmp-XXXX-pack' in the .git/objects/pack folder. All other threads I can find are about non-hidden files with tmp_ prefix in the same folder. The hidden ones are also clearly created during the repack action and it's possible that these get stuck as well. I can't confirm whether that's still possible in git 2.3.0 (which I've updated to since), but at least the disk space requirement doesn't seem to have changed in this newer version - it still can't complete gc/repack. By deleting these .tmp-files I was able to recover my last 4GB and git still seems to behave fine afterwards - your results may vary though, so please make sure you have a backup before doing this. Finally, even 4GB wasn't enough to repack with gc --agressive. My .git folder is 1.1GB after the cleanup, my entire repository is 1.7GB. So 2x the size of your repository is possibly not enough for git gc, even with the aggressive option (which should save space). So I had to recover more space from elsewhere first.

Here is the command I used to clean up (again, have backups!):

git gc --aggressive --prune=now || rm -f .git/objects/*/tmp_* && rm -f .git/objects/*/.tmp-*
Michel Müller
  • 5,535
  • 3
  • 31
  • 49
  • 2
    Interesdting feedback, +. But I confirm `git gc` alone isn't enough. `git repack -Ad` and `git prune` are required as well, as documented in http://stackoverflow.com/a/28563637/6309. – VonC Feb 28 '15 at 09:32
  • 1
    From this answer I will only stress *please make sure you have a backup before doing this*. Removing the files `.git/objects/*/.tmp-*` corrupted my repository beyond repair: all git commands now give me `fatal: bad object HEAD.`, `git fsck` reports plenty of missing blobs, and trying to clone this broken repo to somewhere else did not work. I believe I just lost my whole git history. – unagi Mar 04 '19 at 10:38
  • What's the value of `set -e` here? It has no effect on checked operations, and the only unchecked operation is the last command, and the exit status of that will be used as the overall script's exit status regardless. (See also [BashFAQ #105](https://mywiki.wooledge.org/BashFAQ/105#Exercises) discussing why `set -e` makes scripts' behavior harder for either authors or readers to predict). I'm also not sure why you'd want to mask the `git gc` failing by returning a successful exit status if the first `rm` succeeds after aforementioned `gc` fails. – Charles Duffy Nov 02 '22 at 22:28
  • @CharlesDuffy: Agree, to use use this as a script has robustness issues. the `set -e` is just there because I use it as preamble in every script. I don't like the alternative of *not* having it and then forgetting to check a return code after every operation either. So to me it just comes down to bash being flawed as a scripting language. will change the answer though, thanks for feedback. – Michel Müller Nov 03 '22 at 07:48
7

Similar scenario (about 2.3G available), except git gc itself would also fail with fatal: Unable to create '/home/ubuntu/my-app-here/.git/gc.pid.lock': No space left on device

What worked was to git prune first, and then run the gc.

Sam
  • 2,540
  • 2
  • 23
  • 24
2

I had an instance of this problem. I was able to free up a considerable amount of disk space, but of course, that didn't solve the problem of what to do about the .tmp-* files. I ran git fsck and the Git repository wasn't damaged in that way.

I did the conventional pack-and-garbage-collect operations

git repack -Ad
git prune

but that didn't remove the .tmp-* files, though it would have ensured that all of the necessary objects were in the standard pack-* files if they needed to be copied from transient files left over from the crashed Git processes in the past.

Eventually I realized that I could safely move the .tmp-* files to a scratch directory, then run git fsck to see if what remained in the .git directory was complete. It turned out that it was, so I deleted the scratch directory and the files it contained. If git fsck had reported problems, I could have moved the .tmp-* files back into the .git directory and researched another solution.

user2864482
  • 221
  • 1
  • 4
1

The hidden ones are also clearly created during the repack action and it's possible that these get stuck as well

The way "git repack"(man) created temporary files when it received a signal was prone to deadlocking, which has been corrected with Git 2.39 (Q4 2022).

See commit 9b3fadf (23 Oct 2022), and commit 1934307, commit 9cf10d8, commit a4880b2, commit b639606, commit d3d9c51 (21 Oct 2022) by Jeff King (peff).
(Merged by Taylor Blau -- ttaylorr -- in commit c88895e, 30 Oct 2022)

repack: use tempfiles for signal cleanup

Reported-by: Jan Pokorný
Signed-off-by: Jeff King

When git-repack(man) exits due to a signal, it tries to clean up by calling its remove_temporary_files() function, which walks through the packs dir looking for ".tmp-$$-pack-*" files to delete (where "$$" is the pid of the current process).

The biggest problem here is that remove_temporary_files() is not safe to call in a signal handler.
It uses opendir(), which isn't on the POSIX async-signal-safe list.
The details will be platform-specific, but a likely issue is that it needs to allocate memory; if we receive a signal while inside malloc(), etc, we'll conflict on the allocator lock and deadlock with ourselves.

We can fix this by just cleaning up the files directly, without walking the directory.
We already know the complete list of .tmp-* files that were generated, because we recorded them via populate_pack_exts().
When we find files there, we can use register_tempfile() to record the filenames.
If we receive a signal, then the tempfile API will clean them up for us, and it's async-safe and pretty battle-tested.

And:

repack: drop remove_temporary_files()

Signed-off-by: Jeff King

After we've successfully finished the repack, we call remove_temporary_files(), which looks for and removes any files matching ".tmp-$$-pack-*", where $$ is the pid of the current process.
But this is pointless.
If we make it this far in the process, we've already renamed these tempfiles into place, and there is nothing left to delete.

Nor is there a point in trying to call it to clean up when we aren't successful.
It's not safe for using in a signal handler, and the previous commit already handed that job over to the tempfile API.

It might seem like it would be useful to clean up stray .tmp files left by other invocations of git-repack.
But it won't clean those files; it only matches ones with its pid, and leaves the rest.
Fortunately, those are cleaned up naturally by successive calls to git-repack; we'll consider .tmp-*.pack the same as normal packfiles, so "repack -ad", etc, will roll up their contents and eventually delete them.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250