0

We have buitl a test suite that always worked on local ubuntu machines and on ubuntu machines on github actions. Since the latest additions of unit tests the number of unit tests of one package grew quite substantially and it starts to fail with a too many open files error message:

{"Time":"2021-10-22T13:24:15.784106437Z","Action":"output","Package":"github.com/MyCompany/Product/src/interface/service/user","Test":"Test_userService_MethodTest/Case_1","Output":"2021/10/22 13:24:15 too many open files\n"}

Two strange things appear: On Windows there is no such problem. And if I comment out half of the unit tests in the package mentioned in the error it does not happen either.

The number of unit tests in the affected package is 245 for now and it approximately stops working at around or so.

Heikkisorsa
  • 740
  • 9
  • 31
  • 2
    The error says you are running into a limit for open file descriptors, so you need to figure where you are not releasing these in your tests. Different systems have different limits, even within unix-like OSs. You could increase the ulimit, but that is only putting off the problem until you reach it again next time. – JimB Oct 22 '21 at 14:21
  • it can happen if you are creating a lot of goroutines and each of them is opening a file descriptor (like even a network connection creates a fd) – prembhaskal Oct 22 '21 at 14:56
  • Thanks for the comments. For this particular package we mock a lot of http responses from a third party server. However, we do not open files and we do not use stuff like `httptest`. All the used interactions are mocked as well. – Heikkisorsa Oct 22 '21 at 15:24
  • 1
    Did you take a look at this thread on SO: https://stackoverflow.com/questions/32325343/go-tcp-too-many-open-files-debug ? There are some good insights that might help. – GuiFalourd Oct 22 '21 at 16:44
  • 1
    Are yo running your tests in parallel? If not, then you are opening _something_ and failing to close it, bit we cannot tell what that is without a [mre] – JimB Oct 22 '21 at 19:36
  • @GuiFalourd I worked through all the possible solutions in your link but I could not make it disappear. – Heikkisorsa Oct 25 '21 at 08:40

2 Answers2

1

I think you need to change your max file descriptors. This same problem occurred in many of the development VM's before and needed to change the file descriptors max, not anything with inotify settings.

Note ulimit != ulimit -n

➜  cmd git:(wip-poop) ✗ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       1418
-n: file descriptors                4864
S H A S H A N K
  • 118
  • 2
  • 23
  • 1
    We tried already to change these. However, this cannot be a solution since it is just postponing the problem. What if we have a package with 10 times the amount of unit tests? – Heikkisorsa Oct 23 '21 at 05:15
0

In our test setup we had the config injected by a separate viper instance for each subtest which opened some files everytime. Centrally defining one viper instance and then pass the config down into the tests did the trick.

Heikkisorsa
  • 740
  • 9
  • 31