3

I've been getting ENOMEM errors in my production Node.js (v12) app for a couple of months.

This issue has been particularly hard to diagnose because my heap is always in the 100MB range and doesn't increase over time. Hence a node memory leak is unlikely.

However, some times the VIRT memory usage (as reported by top cmd) goes as high as 15-20GB and that's when node starts to throw some "spawn: enomem" errors.

As far as I see something is making node reserve lots and lots of virtual memory even though the heap is stable at 100MB. And I can't diagnose what it is since the node debuting tools only allow me to inspect the heap.

I've spent close to 100hrs on this problem so any insight / help would truly be appreciated !

I tried:

  • Manualy setting --max-old-space-size
  • Increasing RAM on the server
  • Adding SWAP to my server and allowing docker to use that SWAP
  • Inspecting the node heap with chrome dev tools
  • Setting a memory limit for my docker containers
  • Moving to a different process manager (pm2 instead of supervisord)
  • upgrading all my packages and using the latest LTS version of node
  • Reading and understanding all the google results for node ENOMEM
  • Understanding how Node manages its memory pool, especially regarding heap and GC.
  • Diving into linux kernel concepts of virtual, reserved, swap and shared memory.

Thanks for your help !

user1323873
  • 41
  • 3
  • I'm facing the same problem... In case of some solution, I'll text you here. – Rogério Oliveira Jul 30 '20 at 12:15
  • Im also running into this, I am generating large mp4 files with FFMPEG spawning, but also deleting them when I've finished using them, the problem is Memory Usage (docker stat) keeps accumulating until ENOMEM occurs. I've also noticed /var/lib/docker/overlay2/LONG-HASH/diff and /var/lib/docker/overlay2/LONG-HASH/merged are holding onto my .mp4 files. Skimming through docker documentation they say not to touch /var/lib/docker files so I won't but still have no resolution for this and this does seem very difficult to debug. Will also try to post a solution if I ever come across it. – Ariel Frischer Aug 01 '20 at 20:54

1 Answers1

1

We managed to solve the problem. Keep in mind the solution might is specific to our own project.

We manage to reproduce the error consistently when using the package "imagemin" with "imagemin-jpegtran" & "imagemin-pngquant" on a large array (20k+) of files, and waiting for all promises to finish with a promise.all(). The VIRT mem usage surged to 20GB+ and was never freed after execution.

The solution for us, was to use plimit to limit the number of concurrent promises to 5 or 10. On the same number of images (20k+) the VIRT mem usage never rose, neither did RES, and we stopped getting ENOMEM errors.

Hope this helps someone out there

user1323873
  • 41
  • 3