5

I'm upgrading a secondary replica set member to wiredTiger. I have upgraded it from MongoDB 2.6.3 to 3.0.4 and changed the storage engine to wiredTiger. Now it is resyncing all data from the primary. At some point the following error is received, and the process starts all over again:

2015-07-22T13:18:55.658+0000 I INDEX [rsSync] building index using bulk method

2015-07-22T13:18:55.664+0000 I INDEX [rsSync] build index done. scanned 1591 total records. 0 secs

2015-07-22T13:18:56.397+0000 E STORAGE [rsSync] WiredTiger (24) [1437571136:397083][20413:0x7f3d9ed29700], file:WiredTiger.wt, session.create: WiredTiger.turtle: fopen: Too many open files

2015-07-22T13:18:56.463+0000 E REPL [rsSync] 8 24: Too many open files

2015-07-22T13:18:56.463+0000 E REPL [rsSync] initial sync attempt failed, 9 attempts remaining

The same machine was previously running 2.6.3 version without any open file limits issues. I'm aware that wiredTiger might be creating much more files, so it must be it, but does it keep them all open simultaneously ?

For reference:

cat /proc/sys/fs/file-max

10747371

In /etc/init.d/mongod the configuration is:

ulimit -n 64000

According to the documentation it seems that mongo holds a file descriptor for every data file. As in wiredTiger this results in a file for each collection + a file for each index, according to a calculation for our usecase, can add up to over 700K.

So I can change the ulimit to 700000 or higher, but I'm wondering whether this is the most correct solution, and what alternatives exist.

Baruch Oxman
  • 1,616
  • 14
  • 24
  • You need to check the limits imposed on the process itself (not necessarily your system wide setting). There is a handy script here that does this for you: http://docs.mongodb.org/manual/reference/ulimit/#proc-file-system – Adam Comerford Jul 23 '15 at 19:44
  • I checked this as well - currently my /etc/init.d/mongod script comes with 64000 as the limit. I believe this is not enough - as with wiredTiger there's a separate file per each collection + each index, so for a DB of 100 collections and around 10-20 indices per collection on average, I'm getting around 1000-2000 files per DB, and we're talking about hundreds of DBs. – Baruch Oxman Jul 23 '15 at 19:52

1 Answers1

4

WiredTiger will clean up open file descriptors based on how long they have been idle, but during heavy activity across a large number of collections and indexes you are going to end up bounded by the ulimit on open files.

So, yes, basically you need to increase the limit until you no longer hit the problem, stick with MMAPv1, or consolidate some collections. I would also recommend filing a feature request outlining your use case with your sample numbers for a way to prevent this type of pattern (more than one collection per file for example).

Adam Comerford
  • 21,336
  • 4
  • 65
  • 85