0

I'm using Ansible with a Windows host (VM). I'm trying to git clone a large content (actually using git lfs). It's around 15Go in total

I'm using git for windows (https://github.com/git-for-windows/git/releases/tag/v2.37.2.windows.2).

I have the following Ansible task:

- name: Clone repository
  ansible.windows.win_command: >
    <path/to/git.exe> clone <git-repo>

The problem is that it always fails with the error:

fatal: Out of memory, realloc failed

I tried to increase the windows VM memory, or to play with git config settings, as explained in several posts, for example:

But none of these solutions work for me. It always failed at the same points:

"Filtering content:  48% (440/902), 2.11 GiB | 6.70 MiB/s",
"fatal: Out of memory, realloc failed",
"Error downloading object: xxxxxxx: Smudge error: Error reading from media file: write /dev/stdout: The pipe is being closed.: write /dev/stdout: The pipe is being closed.",
"",
"Errors logged to 'xxxx.log'.",
"Use `git lfs logs last` to view the log.",
"warning: Clone succeeded, but checkout failed.",
"You can inspect what was checked out with 'git status'",
"and retry with 'git restore --source=HEAD :/'"

The weird things is that this error only happen using Ansible. If I connect to the VM and manually run exactly the same git clone command, everything goes well.

Do you have any idea to help me? Thanks

iAmoric
  • 1,787
  • 3
  • 31
  • 64
  • You might try splitting this into two steps: (1) clone without LFS installed, so that you get "pointer files" instead of real ones (use `--no-checkout` to skip the useless checkout). (2) install LFS and poke it into downloading the real files. It's possible that using `--no-checkout` alone first might avoid the problem, but doing it in two steps means that Git can run without the Git-LFS wrappers hogging memory, and then Git-LFS can run with a less-memory-intensive Git operation. – torek Sep 06 '22 at 13:56
  • This is a pure git checkout problem. Removing ansible tag. – Zeitounator Sep 06 '22 at 16:48
  • I understand the problem comes from `git checkout`. However, it only happens when I use Ansible. I don't have this issue when I run directly on the host – iAmoric Sep 07 '22 at 06:22

1 Answers1

1

I experienced the same behaviour in a similar situation. I was trying to do a reference clone for a repository that's originally over 100GB and also contains LFS files. The out of memory was only seen when trying to execute the clone via Ansible's win_shell.

What worked for me was using the Ansible win_shell module and increasing the MaxMemoryPerShellMB to a large enough value. I figured out the value by doubling the memory until it was larger than the default value but smaller or equal to the actual memory limits of my machine.

Including a snippet of what tasks you'll need to resolve this problem successfully:

---
- name: Increase MaxMemoryPerShellMB
  win_shell: 'Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB <value>'

- name: Clone the repository
  win_shell: '<git clone command>'
  args:
    executable: cmd

Additionally, if it's possible for your usecase, I'd recommend skipping the clone of LFS files and fetching the files that you need afterwards. Refer to the git config commands in 'How to clone/pull a git repository, ignoring LFS?'. It's subtle but the answer seems to indicate that GIT_LFS_SKIP_SMUDGE variable is obsolete for Windows. I'm still trying to find a proper piece of documentation for this.

I hope this helps you!

0kB
  • 11
  • 1