0

I've created a simple go script: https://gist.github.com/kbl/86ed3b2112eb80522949f0ce574a04e3

It's fetching some xml from the internet and then starts X goroutines. The X depends on file content. In my case it was 1700 goroutines.

My first execution finished with:

$ go run mathandel1.go 
2018/01/27 14:19:37 Get https://www.boardgamegeek.com/xmlapi/boardgame/162152?pricehistory=1&stats=1: dial tcp 72.233.16.130:443: socket: too many open files
2018/01/27 14:19:37 Get https://www.boardgamegeek.com/xmlapi/boardgame/148517?pricehistory=1&stats=1: dial tcp 72.233.16.130:443: socket: too many open files
exit status 1

I've tried to increase ulimit to 2048.

Now I'm getting different error, script is the same thou:

$ go build mathandel1.go 
# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: flushing $WORK/command-line-arguments/_obj/exe/a.out: write $WORK/command-line-arguments/_obj/exe/a.out: file too large

What is causing that error? How can I fix that?

Marcin Pietraszek
  • 3,134
  • 1
  • 19
  • 31
  • Use a pool of goroutines to issue the requests instead of one goroutine per request. See https://stackoverflow.com/questions/31527948/too-many-open-files-when-making-http-requests and others for info on how. – Charlie Tumahai Jan 27 '18 at 15:28

1 Answers1

1

You ran ulimit 2048 which changed the maximum file size.

From man bash(1), ulimit section:

If no option is given, then -f is assumed.

This means that you now set the maximum file size to 2048 bytes, that's probably not enough for.... anything.

I'm guessing you meant to change the limit for number of open file descriptors. For this, you want to run:

ulimit -n 2048

As for the original error (before changing the maximum file size), you're launching 1700 goroutines, each performing a http get. Each creates a connection, using a tcp socket. These are covered by the open file descriptor limit.

Instead, you should be limiting the number of concurrent downloads. This can be done with a simple worker pool pattern.

Marc
  • 19,394
  • 6
  • 47
  • 51
  • I've run both `ulimit -f X` and `ulimit X` ;). Thanks, for clarification. Maybe you could also review the script, to earn that accepted answer? ;) – Marcin Pietraszek Jan 27 '18 at 15:14
  • There are 1750 items in the geek list, so you're launching 1750 goroutines each performing an http get each using a tcp socket. And all of those are being launched almost at the same time (probably faster than the first `get` returns anyway). The default file descriptor limit is 1024. You should rate limit your downloads. – Marc Jan 27 '18 at 15:20
  • Is there any convinient way to throtle the running gorutines? Let's assume that I'd like to start 1kk of them and have always at most 1k running. – Marcin Pietraszek Jan 27 '18 at 15:22