error MSB4166: Child node exited prematurely. Shutting down

Question

Sometimes my build fail with this error.

 0>MSBUILD : error MSB4166: Child node "3" exited prematurely. Shutting down.

It seems to be completely random and I've not been able to reproduce it at will. I'm running VS2010 Win7 x64 MSBuild 4.0 but this problem seems to be platform and OS independent. I'm building solutions in parallel (/m switch + BuildInParallel=True) and I don't want to disable this feature because I'm compiling application containing 800+ projects . Any idea how to solve it?

EDIT: When I installed .NET 4.5 developer preview, error logging was improved in the MSBuild 4.5 and now the error string looks like this:

error MSB4166: Child node "3" exited prematurely. Shutting down. Diagnostic information may be found in files in the temporary files directory named MSBuild_*.failure.txt

I can find error log file in the Temp folder. This is content of MSBuild_*.failure.txt file:

System.InvalidOperationException: BuildEventArgs has formatted message while serializing!
   at Microsoft.Build.Framework.LazyFormattedBuildEventArgs.WriteToStream(BinaryWriter writer)
   at Microsoft.Build.Framework.BuildMessageEventArgs.WriteToStream(BinaryWriter writer)
   at Microsoft.Build.Shared.LogMessagePacketBase.WriteToStream(INodePacketTranslator translator)
   at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.PacketPumpProc()

I'm having the same troubles. I have also seen out-of-memory exceptions related to this error. Restricting it to one simultaneous build doesn't help; it still errors out: [See the screenshot here](http://i.stack.imgur.com/FSyuG.png); the error occurs precisely when the memory consumption exceeds the physical available memory. It had a few close brushes with death and then it hit the maximum and died, taking down Outlook and Process Explorer with it, offering up a JIT debugger that won't start, and releasing MSBuild.exe to become a zombie process sitting on memory until I manually kill it. — Kevin Vermeer, Nov 01 '11 at 21:54
Strange thing is that I'm using 64bit MSBuild on 64bit Win7 laptop with 4GB of physical and "unlimited" virtual RAM. MSBuild process is using about 1GB of RAM (1,5GB peak). — Ludwo, Nov 02 '11 at 09:43
I'm using 32bit MSBuild on a 32-bit WinXP desktop with 2GB of physical and similarly unlimited virtual RAM. The weird thing is that the crash occurs when physical RAM is completely used up. It's like I've got zero virtual memory! — Kevin Vermeer, Nov 02 '11 at 11:30
But it is not true. I checked it and it failed on this issue if it have 800MB of free physical memory available. — Ludwo, Nov 03 '11 at 20:01

score 14 · Answer 1 · answered Feb 13 '19 at 14:32

14

After a lot of time and research trying to fix this problem I found a solution that worked for me. I am using msbuild with /m and /p:BuildInParallel=true and the build was failing in the CI server. It was always failing in the second componen with the error:

error MSB4166: Child node "3" exited prematurely. Shutting down.

Adding /nodeReuse:false fixed the problem.

This is a good reference: https://blogs.msdn.microsoft.com/msbuild/2007/04/16/node-reuse-in-multiproc-msbuild/

answered Feb 13 '19 at 14:32

Fabio Cirone

361
3
5

3

Jenkins even stopped freezing every building at the end. Two birds with one stone. `MSBUILDDISABLENODEREUSE=1` or `-nodeReuse:false` must be on a build machine. – it3xl Jun 28 '19 at 17:14
A shorter syntax /nr:false also stated in https://blogs.msdn.microsoft.com/msbuild/2007/04/16/node-reuse-in-multiproc-msbuild/ – Wasabi Dec 19 '19 at 14:05
Be aware that the problem can happen even with that flag set to false: https://github.com/microsoft/msbuild/issues/3140 and https://github.com/microsoft/msbuild/pull/3158 if you are using MSBuild versions before April 2018. – Carlos Quintero Jul 16 '20 at 12:01
Found the following related open issue: https://github.com/dotnet/msbuild/issues/455 – Joerg S May 11 '22 at 09:22

score 4 · Answer 2 · answered Nov 02 '11 at 14:56

As discussed in the exchange of comments to the question:

Strange thing is that I'm using 64bit MSBuild on 64bit Win7 laptop with 4GB of physical and "unlimited" virtual RAM. MSBuild process is using about 1GB of RAM (1,5GB peak). – Ludwo 4 hours ago

I'm using 32bit MSBuild on a 32-bit WinXP desktop with 2GB of physical and similarly unlimited virtual RAM. The weird thing is that the crash occurs when physical RAM is completely used up. It's like I've got zero virtual memory! – Kevin Vermeer 3 hours ago

Yes, it seems like MSBuild is not using virtual memory :) – Ludwo 2 hours ago

It did seem like MSbuild wasn't using virtual memory. I did some tests (starting a bunch of programs) and it seemed like nothing was using virtual memory. I did some searches which lead me to check

Control Panel -> System -> Advanced -> Performance -> Advanced -> Virtual Memory

and found that there exists a setting which limited my virtual memory size system-wide. I had imagined that virtual memory to be effectively infinite, or, more precisely, 4 GB for each process on 32-bit XP. I wasn't approaching this limit. However, my virtual memory space was limited to...0MB. Not cool, whoever or whatever did that.

I changed this to allocate a minimum of 1024 MB and a maximum of 4096 MB of virtual memory. I added the "Virtual size" column in Process Explorer, which, together with the "System Commit" graph, demonstrates that I now use more memory than the amount available in the physical RAM sticks.

This fixed my problems. Unfortunately, my system grinds to a near-halt whenever it tries to page any memory, but that's better than a crash. I did re-enable parallel builds; it parallelizes and uses lots of CPU while I have RAM left (which is true for most of the files) and dips to 1% of CPU usage when I have no more RAM. When those files are done, speed is restored.

@Ludwo - That's a good thing, and there are certainly many possible causes for this sort of error, but have you tried setting it manually? — Kevin Vermeer, Nov 02 '11 at 15:49
It is not VM settings issue. I have 800MB of free memory. Now I will check if is it caused by 32bit extensions or not... — Ludwo, Nov 03 '11 at 20:03

score 2 · Answer 3 · answered Jul 30 '17 at 02:18

2

In my case, the answer was to update Antlr. Obviously, this only applies if you are using Antlr in your project.

answered Jul 30 '17 at 02:18

William Jockusch

26,513
49
182
323

score 2 · Answer 4 · answered Jan 30 '19 at 11:46

We got the same msbuild error, and in the log file there was

UNHANDLED EXCEPTIONS FROM PROCESS 10260:
=====================
05/01/2019 18:41:55
System.IO.IOException: Pipe is broken.
  at System.IO.Pipes.PipeStream.WinIOError(Int32 errorCode)
  at System.IO.Pipes.PipeStream.BeginWriteCore(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback callback, Object state)
  at System.IO.Pipes.PipeStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count)
  at System.IO.Pipes.PipeStream.Write(Byte[] buffer, Int32 offset, Int32 count)
  at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent localPacketAvailable, AutoResetEvent localTerminatePacketPump)
===================

We were able to resolve the issue by disabling the msbuild-node reuse feature by setting the environment variable MSBUILDDISABLENODEREUSE=1.

score 2 · Answer 5 · answered May 15 '22 at 17:27

In case someone still runs into this error and none of the other answers work, for me, the only way to solve it reliably, was to set -maxCpuCount:1 (-m:1) (MSBuild command line reference).

This was with dotnet core 6.0.300 on Debian 10 (buster).

Also for reference: if anyone doesn't know how the MSBuild flags are applied when using the dotnet cli, they can be appended to many commands(1) which use MSBuild internally (with either - or /), for example:

# shell
dotnet restore "example/example.csproj" -maxCpuCount:1
dotnet build "example/example.csproj" /maxCpuCount:1 /nodeReuse:false

(1) Exceptions seem to be commands that combine other dotnet * commands, like dotnet run; most of this can be found at the dotnet build reference.

score 1 · Answer 6 · answered Oct 31 '11 at 07:22

1

You could be running out of memory, causing one of the build sub processes to fail - does it fail less if you use /m:2 to restrict it to two concurrent builds? (assuming you have more than 2 cores)

Or, if you can borrow some RAM from another machine, or increase your swap size, does it happen less often when you have more memory installed on your build machine?

answered Oct 31 '11 at 07:22

David Hogan

549
6
6

I have only 2 cores. My build sometimes fail on out of memory exception. I will do some investigation and I will let you know... – Ludwo Nov 01 '11 at 09:01
I have a lot of free memory and my failed again. It should be somehow related to 32bit process memory limitation because my build is using more than 2GB of RAM. But I don't see how it should be possible because my MSBuild process is 64bit. – Ludwo Nov 03 '11 at 20:08

score 1 · Answer 7 · answered Nov 05 '11 at 04:47

1

Perhaps it's the build-equivalent of a race condition?

http://blogs.msdn.com/b/msbuild/archive/2007/04/26/building-projects-in-parallel.aspx

If you are using an ordinary Reference tag for a dependency on the output of another project that is part of the build (rather than a ProjectReference tag) you could be getting a situation where usually project X completes before project Y (which depends on project X's output) but occasionally they build concurrently, in which case the output of X would not exist when Y went to look for it, causing Y to fail. I can't find anything about what sort of error output MSBuild gives in that situation though (and don't have a readily available way to test it just now), so that may not be it.

Still the inconsistency in the outcomes (often successful, occasionally fails) leads me to suspect that something like that may well be the cause.

answered Nov 05 '11 at 04:47

MikeBMcL

414
3
6

No. I have huge number of projects in multiple solutions. I use my merge task for merging solutions and I create one big solution before every build to be able to build all projects in parallel. In merge solutions task I'm converting all references into project references if possible. So my project files are updated after solutions merge as described in example2 in your linked article. – Ludwo Nov 05 '11 at 08:01
+1 for your answer because you indirectly explained me why only one instance of MSBuild is active while other instances are idle. In the discussion I found link to [this bug](https://connect.microsoft.com/VisualStudio/feedback/details/635800/msbuild-exe-appears-to-hang-when-maxcpucount-is-used-and-a-large-number-of-project-references-exist#details). It is fixed in future MSBuild release after v4.0. I have to split my projects into smaller parts to improve msbuild performance :( – Ludwo Nov 05 '11 at 08:06
@Ludwo - If it helps, I had 8 projects in my solution when I found and fixed this problem. They were split into some 800 files and 16,000 lines of code; about 40% of this was contained in one project. – Kevin Vermeer Nov 08 '11 at 18:07
Thanks! So this is normal when this big project was compiled first and all other dependent projects were waiting for it. – Ludwo Nov 08 '11 at 20:10

score 0 · Answer 8 · answered May 17 '19 at 17:03

Just wanted to add my case and resolution for those who weren't able to resolve this issue with the other responses here.

For me, it was because I had inadvertently added a project reference to a .NET Framework 4.6.1 project to one of my .NET Standard 2.0 projects. As of this writing, and VS 2017 15.9.12, you may barely notice this as a "yellow triangle" symbol shown in Solution Explorer in your References that something might be wrong. I had a solution with several of projects, and somewhere in the middle of Build Solution, it would fail with this "Child Node X exited prematurely", on random projects, and never consistent where it failed.

Once I traced the mistake to the offending .NET Framework 4.6.1 referenced project, I converted this .NET Framework 4.6.1 project to .NET Standard 2.0, and all of these Child Node X errors went away.

Hope this helps anyone reading.

score 0 · Answer 9 · edited Apr 25 '23 at 15:22

I was able to resolve this issue in VS 2017 by executing the below commands from the Visual Studio Command prompt.

gacutil /i "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\Microsoft.Build.Framework.dll"
gacutil /i "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\Microsoft.Build.dll"
gacutil /i "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\Microsoft.Build.Engine.dll"
gacutil /i "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\Microsoft.Build.Conversion.Core.dll"
gacutil /i "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\Microsoft.Build.Tasks.Core.dll"
gacutil /i "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\Microsoft.Build.Utilities.Core.dll"

error MSB4166: Child node exited prematurely. Shutting down

9 Answers9

Linked