Node.js EMFILE error with increasing traffic

Question

I have very recently received a lot of traffic to my site that runs Node.js. With the increasing traffic it has started to crash a lot, which has not happened before. I get the following error in my log:

{ [Error: connect EMFILE] code: 'EMFILE', errno: 'EMFILE', syscall: 'connect' }
Error: connect EMFILE
    at errnoException (net.js:670:11)
    at connect (net.js:548:19)
    at net.js:607:9
    at Array.0 (dns.js:88:18)
    at EventEmitter._tickCallback (node.js:192:40)

Anyone that have an idea why it crash? And ideas how to solve it?

I'm using Express.js and Socket.io. It runs on Ubuntu.

`EMFILE` error means that the OS is denying your program to open more files/sockets, have a look at: http://stackoverflow.com/q/34588/511300 — stewe, Apr 27 '12 at 18:08
I'd say, this Q&A is definitely worth **MORE** views, even as many as _Hello World_. Developers open concurrent sockets on demand [_without any limitation_](https://nodejs.org/api/http.html#http_agent_maxsockets), however, they don't know the fact that it means to **allow** massive requests for opening sockets to make their application crash and mess up all the things. **None warns this**, even though it may be a serious vulnerability. **Developers had to be informed** this before they learned how to use [`forever`](https://www.npmjs.com/package/forever). — Константин Ван, Feb 03 '18 at 20:30
To prevent your application from crashing by massive connections, firstly, ascertain no resource leakage ― your application not making and keeping unnecessary connections; then secondly, do either, [limit your available concurrent sockets](https://stackoverflow.com/a/34600633/4510033), or [increase the OS's opened-socket limit](https://stackoverflow.com/a/10421706/4510033). — Константин Ван, Feb 03 '18 at 20:47

score 70 · Accepted Answer · edited May 23 '17 at 12:26

70

EMFILE error means that the OS is denying your program to open more files/sockets.

Have a look at: How do I change the number of open files limit in Linux?

edited May 23 '17 at 12:26

Community

1
1

answered May 02 '12 at 21:04

stewe

41,820
13
79
75

14

Increasing your socket limit won't fix a leak though, it'll only keep your app from crashing for a little while longer. – jpillora Aug 14 '13 at 00:10
@jpillora what do you suggest to prevent leak? I've exactly the same problem and I don't want fix it by increasing ulimit because, as you wrote, it's temporarilly fix. – Rob Aug 22 '13 at 11:45
3

@Robert Sometimes increasing the ulimit is actually the solution, like if it was at 256 and you had high traffic, though I'm my case I was repeatedly opening file steams and never closing them, I caught this leak with lsof -p . If that fails you can use izaacs graceful-fs module – jpillora Aug 23 '13 at 15:59
2

Unfortunatelly graceful-fs module is not fix for me. I reach limit because of sockets, not because of opening to much files with `fs` but thanks for help. – Rob Aug 29 '13 at 07:59
1

sockets and files are both file descriptors, ulimit modifies the **file descriptor** limit – jpillora Sep 04 '13 at 03:34
@RobertJagoda were you able to solve the problem with sockets? I couldn't find a solution when dealing with http.get requests, resulting in EMERROR! :( – neeagl Mar 05 '14 at 18:48
@neeagl yes but I've had somewhere else wrong code causing troubles. You can check my thread: http://stackoverflow.com/questions/18377311/socket-io-every-emit-action-create-new-virtual-file If you have to many connections check if there is everything ok in your code. Maybe something creates too many of connections? Check also ulimit on server. Some servers have it unlimited, others have limits. – Rob Mar 10 '14 at 07:13
Yes, found the error. It was indeed creating too many connections inside a setInterval block. I had to put the initializing code outside it. – neeagl Mar 14 '14 at 21:33

score 23 · Answer 2 · answered May 14 '17 at 13:55

EMFILE means error maximum files which indicates that the OS is denying your program to open more file descriptors.

Note that open files in the system include disk files, named pipes, network sockets and devices opened by all processes.

Your operating system specifies the open file limit per process.

To check open file limit of your system use ulimit -a command. Generally on unix like system it is default to 1024.

If the limitation was 1024, means you/process can open maximum 1024 files. if you exceed this limit means open, pipe and dup system calls will fail and yield EMFILE error.

When you get EMFILE it mostly indicates the leak in your code. Increasing the ulimit to a higher value is not a good solution in case there is leak in your program.

You should try to find out the cause of leak. Following can be used to debug in unix like operating system :

lsof meaning list open files command gives the list of open files. Assume your nodeJS program is having leak so to find out total number of file descriptors opened by your program :

lsof | grep node | wc -l
To find out the file descriptors for sockets, use:

lsof -n -i -P | grep node

Using this we can find out where the leak is and then we can correct it.

score 5 · Answer 3 · answered Jan 04 '16 at 21:59

In my case, we already had the ulimit set and kern.maxfiles configuration highest it could go on a Mac.

What fixed it was limiting the number of max sockets globally. In later versions of node the limit is Infinity.

Try adding these lines of code:

var https = require('https');
var http = require('http');

https.globalAgent.maxSockets = 5;
http.globalAgent.maxSockets = 5;

In the case of using the request library, you can configure these settings.

See: https://github.com/request/request#request---simplified-http-client

Here is more on maxSockets: https://nodejs.org/api/http.html#http_agent_maxsockets

Node.js EMFILE error with increasing traffic

3 Answers3

Linked

Related