BindException using Jetty HttpClient

Question

The following exception occurs intermittently when making requests with Jetty's HttpClient:

    java.net.BindException: Address already in use: connect
        at java.base/sun.nio.ch.Net.connect0(Native Method)
        at java.base/sun.nio.ch.Net.connect(Unknown Source)
        at java.base/sun.nio.ch.Net.connect(Unknown Source)
        at java.base/sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
        at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:433)
        at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.connect(AbstractConnectorHttpClientTransport.java:73)
        at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:602)
        at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:579)
        at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:575)
        at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:181)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

Unfortunately, I don't know much about the specific scenario in which this happens.

We are using an Eclipse Distribution of OpenJDK 17.0.4 with Jetty 10.0.11 on a Windows machine.

Does anyone have an idea what the cause could be?

A quick research (e.g. question 1, 2, and 3) suggests that the system is running out of "ephemeral ports", so it's a TCP issue. If that's the case here too, what am I doing wrong, and how can I fix the problem with a Jetty client?

Joakim Erdfelt · Answer 1 · 2022-09-19T19:10:57.833

Most Common Cause

You've specified an HttpClient.setBindAddress(SocketAddress) that forces the client to use the same local InetAddress every time it wants to connect out to a remote server.

Either be careful of how you use the local bind address, or just don't set the local bind address on the HttpClient.

Jetty's HttpClient is essentially calling java.net.Socket.bind(SocketAddress) and that's what is causing your problem.

Most projects do not need a local bind address when initiating an outgoing connection request. If your organization requires it, then you'll have to deal with everything that it entails, namely that if one client application is using that bind address, another client application cannot. Not even 2 clients in the same application can share it.

Common Cause

Your local address (ip + port) space has exhausted all of it's local outgoing ports. This can happen if you are too aggressive with your client socket creation, but lazy with your socket usage and cleanup. (such as opening a connection, making a request, but not reading the response, possibly not even closing properly).

Since you are on Windows, hopefully Windows 10 or better, use the netstat command line to check your open sockets, and pay attention to the TIME_WAIT entries, you likely have run out of local address combos (ip + port) you can use. Check for TIME_WAIT entries, as those are an indication that you have a socket that was closed improperly.

Interesting, I didn't know about setBindAddress. And according to IntelliJ, there are no calls to it in our project, neither from our code nor from any dependencies (and the same applies to the underlying ClientConnector.setBindAddress). Do you have any ideas what else might cause this? — mperktold, Sep 19 '22 at 15:37
We once had a problem because we didn't read all response content. The effect was that at some point, requests timed out because they waited for the previous ones to complete. There was no BindException. I've just simulated this again with 200 requests with responses whose content is never read, and I still get this behavior: timeout, but no BindException. Thanks for the hints regarding `netstat`, we'll try that! — mperktold, Sep 21 '22 at 07:42

BindException using Jetty HttpClient

1 Answers1

Most Common Cause

Common Cause