23

I am running docker build with a limit on the build's memory and CPU. To stay within the build's CPU and memory limits, I am also limiting Node to a heap size of 325 MB. This is the docker build command.

docker build --build-arg NODE_OPTIONS=--max-old-space-size=325 \
             --memory=600m --memory-swap=-1 \ 
             --cpu-period=100000 --cpu-quota=50000 \
             --no-cache --tag farm_app_image:latest --file Dockerfile .

Build Resource Summary

  • Node JS Heap Limit: 325 MB
  • Build Memory: 600 MB with unlimited use of swap files.
  • Build CPU Time: 50%

Despite having a Node heap limit that is below the build memory, and despite having unlimited swap, the npm run build runs out of memory on the react-scripts build step.

Error Ouput

  > react-scripts build                            

  Creating an optimized production build...                                                                                                                                          
EXEC : FATAL error : Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory [/app/FarmLandLasanga/FarmLandLasanga.csproj]                       

  <--- Last few GCs --->                                                                                                                                                             

  [186:0x5e6ca40]   143688 ms: Mark-sweep 319.3 (329.2) -> 319.2 (329.9) MB, 1278.3 / 0.0 ms  (average mu = 0.118, current mu = 0.002) allocation failure scavenge might not succeed 
  [186:0x5e6ca40]   145181 ms: Mark-sweep 320.3 (329.9) -> 320.2 (331.2) MB, 1488.0 / 0.1 ms  (average mu = 0.060, current mu = 0.003) allocation failure scavenge might not succeed 


  <--- JS stacktrace --->                                                                                                                                                            

  ==== JS stack trace =========================================                                                                                                                      

      0: ExitFrame [pc: 0x1391439]                                                                                                                                                   
      1: StubFrame [pc: 0x1316d29]                                                                                                                                                   
  Security context: 0x154b83ac08d1 <JSObject>                                                                                                                                        
      2: /* anonymous */ [0x14bbf66e0a09] [/app/FarmLandLasanga/ClientAppTypeScript/node_modules/webpack-sources/node_modules/source-map/lib/source-node.js:~86] [pc=0xa4979a73d50](t
his=0x1e34cc96a351 <JSFunction SourceNode (sfi = 0xd8bc3128ee1)>,0x307dd6e664f1 <Object map = 0xb3550d64299>)                                                                        
      3: arguments adaptor frame: 3->1                                                                                                                                               
      4:...                                                                                                                                                                          


  Writing Node.js report to file: report.20191116.195427.186.0.001.json                                                                                                              
  Node.js report completed                                                                                                                                                           
   1: 0x9e9f40 node::Abort() [node]                                                                                                                                                  
   2: 0x9ec192 node::OnFatalError(char const*, char const*) [node]                                                                                                                   
   3: 0xb4611e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]                                                                                         
   4: 0xb46499 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]                                                                           
   5: 0xcf3535  [node]                                                                                                                                                               
   6: 0xcf3bc6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]                                                                                            
   7: 0xd003fa v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]                                                              
   8: 0xd01305 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]                                  
   9: 0xd03dac v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]           
  10: 0xcca66b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]                                                
  11: 0x100eb9e v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]                                                                  
  12: 0x1391439  [node]                                                                                                                                                              
  npm ERR! code ELIFECYCLE                                                                                                                                                           
  npm ERR! errno 1                                                                                                                                                                   
  npm ERR! farm-land-lasagna@0.1.0 build: `react-scripts --max_old_space_size=325 build`                                                                                             
  npm ERR! Exit status 1                                                                                                                                                             
  npm ERR!                                                                                                                                                                           
  npm ERR! Failed at the farm-land-lasagna@0.1.0 build script.                                                                                                                       
  npm ERR! This is probably not a problem with npm. There is likely additional logging output above.                                                                                 

  npm ERR! A complete log of this run can be found in:                                                                                                                               
  npm ERR!     /root/.npm/_logs/2019-11-16T19_54_27_392Z-debug.log

Question

What other changes, if any, can I make to the docker build command so that it succeeds with a 600 MB limit?

References

Additional Attempts

  • With --memory=900m the build succeeds.
  • react-scripts --max_old_space_size=325 build inside of Docker fails.
  • react-scripts --max_old_space_size=325 build outside of Docker succeeds.
Shaun Luttin
  • 133,272
  • 81
  • 405
  • 467
  • 2
    What is the purpose of this docker image? Can you make `npm run build` task outside Docker and put only resulting files into Docker? react-scripts build is quite heavy , do you really need this additional dev dependency in a Docker image? – Łukasz Szewczak Jan 11 '20 at 05:13
  • Try these: https://stackoverflow.com/questions/30252905/nodejs-decrease-v8-garbage-collector-memory-usage – jsur Jan 14 '20 at 18:29
  • What is your base image? did you try changing your base image? – ROOT Jan 14 '20 at 18:42
  • 1
    Depending on what your script is doing, i had create success to reduce the memory on a Docker container with changing the base image to ubuntu 18:04 and applying `RUN apt-get -y install libjemalloc1 ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1` – websoftwares Jan 17 '20 at 10:34
  • Maybe the CPU/speed, the async functions don't end and get more memory used because the process don't ends. If you have 100 promises and each promise resolve in 0.001s and all promises start the execution in 0.001s the max of promises you handle at the same time are only 1. But if each promise resolve in 1s you will handle the 100 promises at the same time (and the memory of each-one). Well I suppose~~ – jtwalters Dec 24 '20 at 04:15
  • It's strange that the file name in the stack trace is .csproj, which is a dotnet C# project file. Are you compiling a C# project in your build? If so, that can eat up memory pretty fast. – Patrick Chu Jul 09 '21 at 10:28
  • @PatrickChu That's a good point that I hadn't considered. Yes. This is a .NET Core project that has SPA (Single Page Application) services running. If I recall correctly, those SPA services are responsible for triggering the `npm` scripts. – Shaun Luttin Jul 09 '21 at 19:31
  • Same problem in angular with docker. [This](https://stackoverflow.com/questions/45363771/docker-build-from-dockerfile-with-more-memory) did not help me. The only choice is increase the ram of your builder machine. – JRichardsz Jul 20 '22 at 19:35

3 Answers3

3

I just write a code, to get the total memory on a linux machine, deduct 400mb (to other server resources), and assign it as node memory limit.

  FREE=$(free -m);
  echo "FREE: ${FREE}"
  FREE_FINAL=$(echo "$FREE" | grep -F Mem:  | grep -o "[0-9]*" | grep -o -m1 "^[0-9]*")
  echo "FREE_FINAL: ${FREE_FINAL}"
  MEM_LIMIT=$(($FREE_FINAL-400))
  echo "MEM_LIMIT: ${MEM_LIMIT}"
  echo " - "

  export MEMORY_LIMIT="${MEM_LIMIT:=3500}";

  export NODE_OPTIONS="--max-old-space-size=${MEMORY_LIMIT}"
  echo "NODE_OPTIONS: ${NODE_OPTIONS}"

You can add it to a entrypoint.sh file (that will run on build time, but on each docker start as well), or on a build.sh file (that will get this value to pass to your docker build command as argument.

Tiago Gouvêa
  • 15,036
  • 4
  • 75
  • 81
1

Can you change your base image? I think that can help. This memory leak problem was in NodeJS v10 and v12, just use latest LTS version (v14) and also alpine version of the image.

David Gabrielyan
  • 227
  • 2
  • 12
1

when using a GC language like JS or Java you have the most popular part of the language memory component which is the heap but you also have another part that is being used when running or building an application, that is the stack.

In any way, I would suggest monitoring the docker build memory usage with docker stats or any other monitoring tool you prefer as well as the react-scripts build memory usage if you use the --expose-gc option (in reference to this thread) and check the docker build logs for any spikes in usage. Running the react-scripts build command outside could help to try and determine what memory requirements the build process actually requires and setting it to it plus some buffer in the container after.

It could be that you're limiting your heap size for nothing to half of the container total memory capacity you've set (600MB), try increasing the heap memory capacity while keeping the containers one the same (increase --max_old_space_size while keeping --memory at 600MB).

Anbother option that seems to help in this and that case is to upgrade node base image to node:lts or node:14.16.0 (are the same image as of the time of writing this).

Noam Yizraeli
  • 4,446
  • 18
  • 35