0

I have written a program in golang to make request about 2000qps to different remote ip with local port randomly selected by linux, and close request immediately after connection established, but still encounter bind: address already in use error periodically

what I have done:

  1. net.ipv4.ip_local_port_range is 15000-65535
  2. net.ipv4.tcp_tw_recycle=1 net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_fin_timeout=30

above is sockstat:

sockets: used 1200 TCP: inuse 2302 orphan 1603 tw 40940 alloc 2325 mem 201

I don't figure it out why this error still there with kernel selecting available local port,will kernel return a port in use ?

Peter
  • 29,454
  • 5
  • 48
  • 60
what is what
  • 1,461
  • 2
  • 12
  • 16
  • 1
    You can't. You can only use something rather less than 65535 local ports every two minutes, because of the TIME_WAIT state. If you need more, you're doing something wrong. Look into connection pooling. – user207421 Mar 27 '18 at 09:35
  • @EJP I have set the tcp_tw_reuse option ,why can't I use these port – what is what Mar 27 '18 at 09:38
  • Because of what I have already stated. I'm not going to just repeat myself here. And, incidentally, because of what is already stated in your link, although it is about servers, not clients. – user207421 Mar 27 '18 at 09:42
  • @kostix this range is limited,not very effective，just want to know why so many port in timewait can not reuse with tcp_tw_reuse set – what is what Mar 27 '18 at 09:57
  • @whatiswhat, I've misread your question, sorry. I'd try looking at `SO_REUSEPORT` and/or `SO_LINGER` on your sockets. Start [here](https://stackoverflow.com/a/14388707/720999). The basic idea is to either use `SO_LINGER` to individually make each ephemeral port die faster or pick a pool of, say, 2k ports, and bind each one on the client side with `SO_REUSEPORT` before connecting — effectively zapping any half-dead connection with the same port, if any. – kostix Mar 27 '18 at 10:03
  • 2
    Why are you setting `tcp_tw_recycle`? I’ve never seen advice to try and use this, because it doesn’t work as expected. It has actually been removed from the latest kernels since it’s basically broken. – JimB Mar 27 '18 at 11:03
  • 1
    I would *not* look at SO_LINGER. It causes a 'connection reset' at the peer, and all data in transit to be lost. @kostix – user207421 Mar 28 '18 at 01:05

1 Answers1

1

This is a good answer from 2012:

https://serverfault.com/questions/342741/what-are-the-ramifications-of-setting-tcp-tw-recycle-reuse-to-1#434669

As of 2018, tcp_tw_recycle exists only in the sysctl binary, is otherwise gone from the kernel:

https://github.com/torvalds/linux/search?utf8=%E2%9C%93&q=tcp_tw_recycle&type=

tcp_tw_reuse is still in use as described in the above answer:

https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_ipv4.c#L128

However, while a TCP_TIMEWAIT_LEN is in use:

https://github.com/torvalds/linux/search?utf8=%E2%9C%93&q=TCP_TIMEWAIT_LEN&type=

the value is hardcoded:

https://github.com/torvalds/linux/blob/master/include/net/tcp.h#L120

and tcp_fin_timeout refers to a different state:

https://github.com/torvalds/linux/blob/master/Documentation/networking/ip-sysctl.txt#L294

One can relatively safely change the local port range to 1025-65535.

For kicks, if there were a situation where this client was talking to servers and network under my control, I would build a new kernel with a not-to-spec TCP_TIMEWAIT_LEN, and perhaps also fiddle with tcp_max_tw_buckets:

https://github.com/torvalds/linux/blob/master/Documentation/networking/ip-sysctl.txt#L379

But doing so in other circumstances- if this client is behind a NAT and talking to common public servers- will likely be disruptive.

Jonah Benton
  • 3,598
  • 1
  • 16
  • 27