14

What does git mean by, "unable to migrate objects to permanent storage"?

 Counting objects: 4, done.
 Delta compression using up to 8 threads.
 Compressing objects: 100% (4/4), done.
 Writing objects: 100% (4/4), 956 bytes | 0 bytes/s, done.
 Total 4 (delta 2), reused 0 (delta 0)
 error: failed to push some refs to 'https://git.patrikx3.tk/server-scripts.git'
 To https://git.patrikx3.tk/server-scripts.git
 !  refs/heads/master:refs/heads/master [remote rejected] (unable to migrate objects to permanent storage)
 Done
donjuedo
  • 2,475
  • 18
  • 28
Patrik Laszlo
  • 4,906
  • 8
  • 24
  • 36

2 Answers2

16

You can see the error message "unable to migrate objects to permanent storage" introduced in commit 722ff7f for Git 2.11 in Oct. 2016.

The explanation is:

receive-pack: quarantine objects until pre-receive accepts

When a client pushes objects to us, index-pack checks the objects themselves and then installs them into place.
If we then reject the push due to a pre-receive hook, we cannot just delete the packfile; other processes may be depending on it. We have to do a normal reachability check at this point via git gc.

But such objects may hang around for weeks due to the gc.pruneExpire grace period. And worse, during that time they may be exploded from the pack into inefficient loose objects.

Instead, this patch teaches receive-pack to put the new objects into a "quarantine" temporary directory.
We make these objects available to the connectivity check and to the pre-receive hook, and then install them into place only if it is successful (and otherwise remove them as tempfiles).

The code is:

    /*
     * Now we'll start writing out refs, which means the objects need
     * to be in their final positions so that other processes can see them.
     */
    if (tmp_objdir_migrate(tmp_objdir) < 0) {
        for (cmd = commands; cmd; cmd = cmd->next) {
            if (!cmd->error_string)
                cmd->error_string = "unable to migrate objects to permanent storage";
        }
        return;
    }
tmp_objdir = NULL;

The tmp_objdir_migrate() function comes from commit 2564d99 (still for Git 2.11)

it helps callers create a temporary directory inside the object directory, and a temporary environment which can be passed to sub-programs to ask them to write there (the original object directory remains accessible as an alternate of the temporary one).

As mentioned, this could result from a permission issue (or disk space issue)

Also, using (on the server side) a git 2.10 would likely make that error disappear.


Git 2.13 (Q2 2017) will expand on that quarantine notion:
See commit d8f4481, commit eaeed07, commit 360244a (10 Apr 2017) by Jeff King (peff).
(Merged by Junio C Hamano -- gitster -- in commit 9f1384f, 24 Apr 2017)

git receive-pack man page now includes:

Quarantine Environment

When receive-pack takes in objects, they are placed into a temporary "quarantine" directory within the $GIT_DIR/objects directory and migrated into the main object store only after the pre-receive hook has completed. If the push fails before then, the temporary directory is removed entirely.

This has a few user-visible effects and caveats:

  1. Pushes which fail due to problems with the incoming pack, missing objects, or due to the pre-receive hook will not leave any on-disk data. This is usually helpful to prevent repeated failed pushes from filling up your disk, but can make debugging more challenging.

  2. Any objects created by the pre-receive hook will be created in the quarantine directory (and migrated only if it succeeds).

  3. The pre-receive hook MUST NOT update any refs to point to quarantined objects. Other programs accessing the repository will not be able to see the objects (and if the pre-receive hook fails, those refs would become corrupted).


razor7 proposes in the comments:

In my case, using GitLab from Docker, ran those commands from the host server (not the GitLAb container)

cd /media/data/gitlab/data/git-data/repositories/@hashed 
grep "gitlab-user/repo-name" . -R 
cd /media/data/gitlab/data/git-data/repositories/@hashed/found-folder 
chown -R 998:998 ./hash-repo-folder-name.git/. 

Have in mind that 998 may vary for you.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • 10
    Permissions! The previous guy doing our repositories had a habit of always using root. chmod -R g+w *, fixed the error. – Martlark Jun 22 '17 at 01:50
  • 2
    chmod -R g+w * fix my issue – Java Dude Feb 07 '18 at 12:24
  • 9
    `chown -R git:git`on the `objects` folders (gitlab server: repositories) fixed the issue. some `objects` folders were owned by `root` mysteriously (2 in fact). – MensSana May 03 '19 at 18:20
  • I have the same situation but it looks like whenever I push the new files are created with the wrong permissions each time - is there a way of deciding which permissions they are created with? – Stefan Aug 19 '20 at 09:41
  • @Stefan on checkout, locally, `umask` can help (if your OS supports it): https://stackoverflow.com/a/25441170/6309. But on commit, only 644 or 755 are recorded: https://stackoverflow.com/a/40979867/6309 – VonC Aug 19 '20 at 12:35
  • menssana i am using gitlab, but i did not understand what object repo folder do you refer to, could you please tell me exactly ? – MatteoM Feb 22 '21 at 10:20
  • @MatteoM Anything under the `.git` folder, which includes all the repository objects. – VonC Feb 22 '21 at 14:08
  • In my case, using gitlab from docker, ran this commends form the host server (not the gitlab container) `cd /media/data/gitlab/data/git-data/repositories/@hashed` `grep "gitlab-user/repo-name" . -R` `cd /media/data/gitlab/data/git-data/repositories/@hashed/found-folder` then ran `chown -R 998:998 ./hash-repo-folder-name.git/`. Have in mind that `998` may vary for you. – razor7 Jul 08 '21 at 12:26
  • 1
    @razor7 Thank you for this feedback. I have included your comment in the answer for more visibility. – VonC Jul 08 '21 at 13:13
1

Presumably when it compresses objects it puts them somewhere temporarily, then in a separate action tries to move them to their permanent location. Moving them means copying them from the temporary location to the permanent location, then either deleting them from the temporary location or leaving them to be removed by other processing, possibly at another time. For example, many of the files and directories that get written to the /tmp directory in Linux are customarily left there, until the operating system removes them upon the next reboot (if there ever is a reboot).

The failure looks like a lack of permission to write to the remote location. The failure occurred when git tried to move the objects that it compressed to the remote location https://git.patrikx3.tk/server-scripts.git that you would have specified in the project's configuration. Less likely, you could have specified it in the command or (I think?) in your global configuration.

Aaron Mansheim
  • 425
  • 3
  • 11