0

I have an automated test that tries to clone a bunch of specific revisions from the local Git repository by using the following asynchronous Nim procedure.

proc cloneSpecificRevision(downloadMethod: DownloadMethod,
                           url, downloadDir: string,
                           vcsRevision: Sha1Hash) {.async.} =
  assert vcsRevision != notSetSha1Hash
  display("Cloning", "revision: " & $vcsRevision, priority = MediumPriority)
  case downloadMethod
  of DownloadMethod.git:
    let downloadDir = downloadDir.quoteShell
    createDir(downloadDir)
    discard await tryDoCmdExAsync("git", @["-C", downloadDir, "init"])
    discard await tryDoCmdExAsync("git",
      @["-C", downloadDir, "remote", "add", "origin", url])
    discard await tryDoCmdExAsync("git",
      @["-C", downloadDir, "fetch", "--depth", "1", "origin", $vcsRevision])
    discard await tryDoCmdExAsync("git",
      @["-C", downloadDir, "reset", "--hard", "FETCH_HEAD"])
  of DownloadMethod.hg:
    discard await tryDoCmdExAsync("hg", @["clone", url, "-r", $vcsRevision])

The result is

Downloading /tmp/tlockfile/origins/dep1 using git
    Cloning revision: 8b6ce61df05f4e21ad954f9ddb487eda8fb64f41
  Executing git -C /tmp/nimble_94295/_tmptlockfileoriginsdep1_0.1.0_8b6ce61df05f4e21ad954f9ddb487eda8fb64f41 init
Downloading /tmp/tlockfile/origins/dep2 using git
    Cloning revision: d1ba2f37a1647ca1ed804dfa0c8c73a7de98201e
  Executing git -C /tmp/nimble_94295/_tmptlockfileoriginsdep2_0.1.0_d1ba2f37a1647ca1ed804dfa0c8c73a7de98201e init
    Error:  Resource temporarily unavailable (code: 11)

The used Git version is:

> git --version
git version 2.31.1

The problem happens only on Linux and my tests are passing normally on Windows and macOS systems. What can be the reason for this problem?

bobeff
  • 3,543
  • 3
  • 34
  • 62
  • This message comes from a tool other than Git, since Git doesn't produce output like that. What tool is that, and how does it work if you clone the repository by hand? – bk2204 Jul 19 '21 at 00:17
  • @bk2204 It has to be from Git because I don't use other tools in this test. I added some additional logging with the exact command which is being executed. – bobeff Jul 19 '21 at 00:31
  • @bk2204 The problem happens when I try to execute in parallel two asynchronous Git commands. – bobeff Jul 19 '21 at 00:43

1 Answers1

1

Error 11 on Linux is EAGAIN, Resource temporarily unavailable. The Linux fork system call is probably producing this error because, well, there are not enough resources at the moment.

What's a resource? There are a lot of them, but one of them is memory and another is processes. Both can be "temporarily" used up since programs (processes) run for a while, then exit; when they exit, they release the resources they were using. See also Fork fails with "resource temporarily unavailable". Which resource?, OSError: [Errno 11] Resource temporarily unavailable. What causes this?, and fork() failing with Out of memory error (note that the last one shows errno 12, ENOMEM, but you can get EAGAIN on other Unix-like systems here so I've included it even though you mentioned Linux specifically).

Note that this has nothing to do with Git itself: if you've hit a process limit with fork, it won't matter what you intend to run after the fork returns in the child process.

torek
  • 448,244
  • 59
  • 642
  • 775
  • Well, is there some way to overcome this issue? I started only two simultaneous Git processes. I don't understand how is possible to hit the limit. Even more, the code was working some time ago. – bobeff Jul 19 '21 at 02:29
  • 1
    You'll need to figure out *what the system is running out of* (is it processes? what are your limit settings?) See the other questions. – torek Jul 19 '21 at 02:30
  • Any idea how to determine what the system is running out of? – bobeff Jul 19 '21 at 12:25
  • 2
    Hopefully some of the related questions have ways of finding out. Follow up with them. – torek Jul 19 '21 at 14:39
  • @bobeff You could try [increasing the different limits tweakable through the ulimit command](https://linuxhint.com/linux_ulimit_command/). Try setting everything to unlimited, and if that works, backtrack to see which limit was holding you down (or just leave unlimited for your test suite). – Grzegorz Adam Hankiewicz Jul 20 '21 at 15:05