Maximum number of open filehandles per process on OSX (and how to increase)

Question

EDIT: I now have a solution, but I'd really apprecite a concise description of what the different limits are, i.e. those set by FD_SIZE, launchtl limit files, sysctl -w kern.maxfilesperproc, ulimit etc.)

Can someone help me understand the limits on open filehandles on OSX. ulimit gives me one answer:

$ ulimit -a
...
open files                      (-n) 256

I can't use ulimit to change this, but people suggest using launchctl (e.g. http://usrinapto.wordpress.com/2010/03/06/mac-os-x-10-6-max-open-files-too-many-open-files/)

Using this doesn't change the limit reported by ulimit, though.

However, my application seems to able to open 10k files before crashing, as reported by lsof, e.g.:

$ lsof -p 87599 | wc
10279   92505 1418903

(it crashes somewhere between 10279 and 10305 open files, reliably)

So there are clearly different limits coming in to play. I've also seen talk (on the above link) of FD_SETSIZE.

Can someone explain to me what the different limits are, and how they are set?

In case it's relevant, I'm working on wrapping a C/C++ library for use in Java, using SWIG.

EDIT: I've also tried:

sudo sysctl -w kern.maxfiles=20000

with no success. Also

#define FD_SETSIZE 20000

has no effect.

EDIT: Also tried

launchctl limit maxfiles 20000 20000

with no effect.

EDIT: Solution:

sysctl -w kern.maxfilesperproc=20000

(via http://krypted.com/mac-os-x/maximum-files-in-mac-os-x/)

EDIT: I've written a small program to test this (based on How to increase the limit of "maximum open files" in C on Mac OS X), and found that the max number of open files I can ask for is 10240:

#include <sys/resource.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

struct rlimit limit;
void setLimit( int l );
void getLimit();

int main( int argc, char* argv[] )
{
    getLimit();
    setLimit(10240);
    getLimit();
    return 1;
}

void setLimit( int lim )
{
    limit.rlim_cur = lim;
    limit.rlim_max = lim;
    printf( "Setting limit to %d, %d\n", limit.rlim_cur, limit.rlim_max );
    if (setrlimit(RLIMIT_NOFILE, &limit) != 0) {
    printf("setrlimit() failed with errno=%d\n", errno);
    exit(1);
    }
}

void getLimit()
{
    /* Get max number of files. */
    if (getrlimit(RLIMIT_NOFILE, &limit) != 0)
    {
        printf("getrlimit() failed with errno=%d\n", errno);
        exit(1);
    }
    printf("The soft limit is %llu\n", limit.rlim_cur);
    printf("The hard limit is %llu\n", limit.rlim_max);
}

score 24 · Accepted Answer · answered Mar 21 '11 at 13:27

24

found on http://krypted.com/mac-os-x/maximum-files-in-mac-os-x/:

sysctl -w kern.maxfilesperproc=20000

answered Mar 21 '11 at 13:27

mo-seph

6,073
9
34
35

5

10.7.5, seems doesn't work: `sudo sysctl -w kern.maxfilesperproc=20000` prints: `kern.maxfilesperproc: 20000 -> 20000` - OK, then `ulimit -S -n` prints 1024 after it ??? – yetanothercoder Jun 29 '13 at 21:22
3

@yetanothercoder, from the article linked above: "Once you’ve done this, the kernel itself will have a maximum number of files but the shell might not. And since most processes that will take up this many files are going to be initiated by the shell you’re gonna’ want to increase that. You can change shell limits using the ulimit command. To do so: `ulimit -S -n 2048`" – Jon Aug 27 '14 at 16:41
1

I've tried both of those steps on OS 10.8.5, and ulimit -n is still 256 after a reboot. – Mykle Hansen Oct 09 '14 at 16:51

score 7 · Answer 2 · answered Mar 21 '11 at 13:21

7

To check limits on your system run: 'launchctl limit'. To change max number of open files run: 'launchctl limit maxfiles 400000 unlimited'. To change the setting permanently add to the file: '/etc/launchd.conf' following line:

limit maxfiles 400000
unlimited

(source)

... and why on earth you need so many open files at once?

answered Mar 21 '11 at 13:21

Unfortunately, that doesn't work for me - I'm not sure what limit that sets, but it doesn't let me open more files. – mo-seph Mar 21 '11 at 13:25
As to why - I'm wrapping up a library, and having to abuse it horribly. This will get refactored out, once I can verify it behaves correctly. – mo-seph Mar 21 '11 at 13:26
5

benchmarking tools often need to open hundreds of thousands of files (or sockets, rather). For example, i'm benching a little server with ~100000 open connections. – jbenet Jun 16 '11 at 10:36
You may need to created the /etc/launchd.conf file if it does not already exist. `limit maxfiles 20000 20000` resolved my problems and `ulimit` reflects this. – Elliot Coad Nov 01 '13 at 11:41
I tried creating /etc/launchd.conf on os 10.8.5 and it had no effect; the settings in the file seem to be ignored. I tried the launchctl command you provided, and got this error: "Neither the hard nor soft limit for "maxfiles" can be unlimited. Please use a numeric parameter for both." (Even though 'launchctl limit' says 'maxfiles 256 unlimited' ....) – Mykle Hansen Oct 09 '14 at 16:55
1

As @ElliotCoad mentions, use a numeric, not unlimited value in that file. I use `limit maxfiles 65536 65536` on 10.8.5 and it works fine. – BeeOnRope Apr 08 '15 at 07:00
Thank you BeeOnRope, that worked for me. On MacOS 10.12.4. – Peheje Mar 29 '17 at 08:07
20 years after the C10K problem you just shouldn't ask why anymore. Hundert of thousands is not a lot when you have a macOS computer with 1 TB RAM – Lothar Jun 14 '19 at 09:30
@jbenet ha, did i stumble upon a piece of IPFS history right here? gold. The irony is i came here due to IntelliJ crashing while indexing an app I'm building on IPFS. – Tails Sep 16 '21 at 19:28

Maximum number of open filehandles per process on OSX (and how to increase)

2 Answers2

Linked