1

We recently moved from hosted to private agents, because of reasons that are not relevant to this question. The problem we're having now, is that the private agent runs out of disk space. I've checked why this is the case, and it turns out that for one of the workspaces the agent creates, the .git folder grows to over 20Gb during the day, while the repository is only a few Gb. What can explain this excessive growth?

some extra info:

  • We build from different branches, using the same pipeline (so it re-uses the same workspace)
  • We do not clean the workspace between runs, since this would require is to re-get the entire repository each build, which slows the build. (I understand adding the clean option would solve our problem, but it would also slow down all builds, which we don't want)
  • We used to use fetchdepth: 1 in our pipelines, but we recently removed this, since it is no longer necessary on private agents, since the sources are cached between runs

Edit: to clarify, I'm looking for a way to avoid running out of disk space on the agents, without losing the ability to cache source files.

PaulVrugt
  • 1,682
  • 2
  • 17
  • 40

1 Answers1

0

When I run the same pipeline with different branches, the .git folder size indeed increases.

Then I find that the root cause of this issue could be the pack files in .git/objects/pack.

enter image description here

It will pack the source files, if your source files are large enough, the packaged files will also take up a lot of space.

You could try to use BFG tool or Git command to remove the files.

For more detailed information, you could refer to this ticket: Remove large .pack file created by git

Kevin Lu-MSFT
  • 20,786
  • 3
  • 19
  • 28
  • Thanks @kevinLu-MSFT . But how do I do this on my scale set agent? Our scale set agent refreshes the agents everynight, but the disk space issues already occuring during the day. Do you suggest I add some scheduled task to clean it each hour? And how do I make sure it doesn't run when a build is running to avoid conflicts? – PaulVrugt Nov 12 '20 at 12:31
  • Hi @PaulVrugt. To avoid conflicts, you can try to add a task to clean up the folder after the last step of the build. Then the folder will be cleaned after build completed. It wiil not cause conflict. – Kevin Lu-MSFT Nov 13 '20 at 08:06
  • Yeah but that's exactly what I'm trying to avoid (see my original question). Because cleaning the folder would result in the sources having to be downloaded for each job, significantly slowing down the build. My question was how to prevent out of disk space errors on the scale set without cleaning the sources each build. Your answer helps me understand what is causing the disk space usage, but not how to solve my problem – PaulVrugt Nov 13 '20 at 08:17