Mongodb too many open files in system

Question

our software containing:

centos, nginx, kestrel http server, dotnet core (v2.0.8) application work with mongodb (v3.6.7)

kestrel http server sometimes stop responding, it seems that an error from mongodb driver connection like below logs on 3th line

what is causing this problem ?

i have configured max open files option to 640000 with this topic

https://medium.com/@mshanak/soved-dotnet-core-too-many-open-files-in-system-when-using-postgress-with-entity-framework-c6e30eeff6d1

but nothing fixed please can you help to solve this problem thank you all.

May 17 12:37:36 myserver[14611]: fail: Microsoft.AspNetCore.Server.Kestrel[13]
May 17 12:37:36 myserver[14611]: Connection id "0HLDRQRA00CP4", Request id "0HLDRQRA00CP4:00000001": An unhandled exception was thrown by the application.
May 17 12:37:36 myserver[14611]: MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: Too many open files in system
May 17 12:37:36 myserver[14611]: at System.Net.Sockets.Socket..ctor(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType)
May 17 12:37:36 myserver[14611]: at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateSocket(EndPoint endPoint)
May 17 12:37:36 myserver[14611]: at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__48.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of inner exception stack trace ---
May 17 12:37:36 myserver[14611]: at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__48.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at MongoDB.Driver.Core.Servers.Server.<GetChannelAsync>d__30.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at MongoDB.Driver.Core.Operations.FindOperation`1.<ExecuteAsync>d__111.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at MongoDB.Driver.OperationExecutor.<ExecuteReadOperationAsync>d__3`1.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at MongoDB.Driver.MongoCollectionImpl`1.<ExecuteReadOperationAsync>d__83`1.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at MongoDB.Driver.MongoCollectionImpl`1.<UsingImplicitSessionAsync>d__91`1.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at MongoDB.Driver.IAsyncCursorSourceExtensions.<FirstOrDefaultAsync>d__5`1.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at MYData.UserCollection.<GetByIdAsync>d__2.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at MYAPI.Controllers.PostController.<GetForExplore>d__8.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__12.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__10.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__14.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.<Invoke>d__6.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
May 17 12:37:36 myserver[14611]: --- End of stack trace from previous location where exception was thrown ---
May 17 12:37:36 myserver[14611]: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
May 17 12:37:36 myserver[14611]: at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
May 17 12:37:36 myserver[14611]: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1.<ProcessRequestsAsync>d__2.MoveNext()

score 0 · Accepted Answer · answered May 17 '18 at 14:24

0

If you are using CentOS 6 or higher then you may have to follow the instructions below.

Red Hat Enterprise Linux and CentOS 6 place a max process limitation of 1024 which overrides ulimit settings. Create a file named /etc/security/limits.d/99-mongodb-nproc.conf with new soft nproc and hard nproc values to increase the process limit. See /etc/security/limits.d/90-nproc.conf file as an example

You can find more details regarding ulimit settings here

answered May 17 '18 at 14:24

webmite

575
3
6

I have created that file and set up to 64000. How can test it? – bits8 May 17 '18 at 15:01
details regarding how to check ulimit values and system level limits for different OS's can be found by following the link specified. Remember to restart the mongodb service in order for the changes to be applied. – webmite May 17 '18 at 15:19
For CentOS the following applies. The /proc file-system stores the per-process limits in the file system object located at /proc//limits, – webmite May 17 '18 at 15:22
i have configured those values, will be test in next two day and update here thank you again @webmite. – bits8 May 17 '18 at 16:42
thank you thats config fixed. but my server memory still rising and don't undestanda why. I have build small bash script for watch memory and restart services. Have you any better solution? Thank you. – bits8 May 21 '18 at 10:22
Read the docs about memory diagnostics. [https://docs.mongodb.com/manual/faq/diagnostics/]. It is very common for people to under-estimate how much memory MongoDB will use. – webmite May 21 '18 at 13:31
Thank you, but i am troubling memory usage of dot net core. How can diagnostic that i am using visual studio community for mac is there a way to do that? – bits8 May 21 '18 at 14:48
I will just point you towards another question asking similar details [https://stackoverflow.com/questions/5274566/detecting-memory-leaks-in-asp-net] – webmite May 21 '18 at 15:10
thank you but, i am using dot net core and that question old dated. – bits8 May 21 '18 at 17:39

Mongodb too many open files in system

1 Answers1