0

We've recently gotten an app written in golang that stores information about visits in a database.

We are facing the following error in this legacy code that is already in production:

http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 1s

We believe that this errors are caused by calls to the database that are getting opened, but can't find the point where this calls get stacked.

Is there a way to check if there are database calls opened and where in the code are?

We've seen that the code uses the github.com/astaxie/beego/orm library and the calls are made using o := orm.NewOrm() but we don't know where else to look for.

UPDATE

The too many open files error appears when the total amount of open files for that process (lsof -u myuser | grep "10930" | wc -l) reaches 2048

Also, the output of lsof -u myuser | grep "protocol: TCPv6" throws something like this:

ana     10930 myuser 3405u     sock       0,8      0t0 307230406 protocol: TCPv6
ana     10930 myuser 3407u     sock       0,8      0t0 307230465 protocol: TCPv6
ana     10930 myuser 3408u     sock       0,8      0t0 307231438 protocol: TCPv6
ana     10930 myuser 3427u     sock       0,8      0t0 307234900 protocol: TCPv6
ana     10930 myuser 3441u     sock       0,8      0t0 307236431 protocol: TCPv6
ana     10930 myuser 3446u     sock       0,8      0t0 307237446 protocol: TCPv6
ana     10930 myuser 3457u     sock       0,8      0t0 307239557 protocol: TCPv6

And the number of this sock connections is getting higher and higher.

Does someone know what these connections are and why are they getting opened?

Pedro Piter
  • 81
  • 7
  • Can you check if the database connections are being opened in go-routines ? Look for key `go` keyword in your code. – Auyer Oct 21 '19 at 17:40
  • 1
    Make sure connection pooling is properly configured. Take a look here: https://beego.me/docs/mvc/model/orm.md orm.RegisterDataBase has 2 additional parameters, except standard 1-3 for connection configuraiton, it has also maxIdle connections and maxConn (parameter 4 and 5, respectively). Make sure it's properly set. Also each RDBMS has possibility to check active connections. For instance, postgresql: select * from pg_stat_activity where datname = 'YOURDB'; this way you can check if you're hitting connections limit for your backend, if you have access there. – Dmytro Oct 21 '19 at 17:49
  • @Auyer yes, there are go routines like this: `go visitor.TrackEvent(eventPayload, eventPayload.Event)`that open database connections in this way: o := orm.NewOrm() visitor := Visitor{TrackingCode: eventPayload.TrackingCode, Key: eventPayload.VisitorKey} err := o.Read(&visitor, "tracking_code", "visitor_key") – Pedro Piter Oct 21 '19 at 22:35
  • @Dmytro Check my answer below. Hope it helps! – Auyer Oct 21 '19 at 22:37

2 Answers2

0

This could be caused by too many connections being opened in various go-routines. If this is the case, configure a worker pool to limit the amount of open files possible : gobyexample worker pools

Check this question "Go, tcp too many open files debug". The accepted answer may give you a fast solution, but it does not fix the main issue.

Auyer
  • 2,693
  • 3
  • 15
  • 32
0

Too many open files are basically related to FD's(File descriptors) of a particular process which are used to store meta information about the process. And there are only limited FD's you can create for a particular process. Check for the output of ulimit -aH command. By default, you can only create 1024 FD's for a particular process. And for every new connection to the DB, a new FD would be created and destroyed when the connection is closed.

You would want to increase this FD value for all the process. This answer should be able to help you with it. https://askubuntu.com/a/1193851/1309493

Abdul Rehman K
  • 11
  • 1
  • 4