1

Working with a Java client/server application and no access to source code.

The client application uses the URLConnection object when making calls to the server. Most of the time it works but sporadically it fails with the following error.

java.net.BindException:Address already in use

When it fails it will do so for a number of attempts and then all of a sudden work fine again.

In an attempt to capture the URL calls, the application was started via the command line and referenced the logging.properties file with the following entry.

 sun.net.www.protocol.http.HttpURLConnection.level = ALL

This generated some good information but there was no reference as to the port it was attempting to open on the client side.

  • Question What entry needs to be added to the logging.properties file to capture the port the client is attempting to use?
Unhandled Exception
  • 1,427
  • 14
  • 30
  • Not sure altering logging levels would work. How do you know that there will be a call to the logger to log the port number? I would maybe approach this by starting the application in debug mode and then attaching a debugger set to stop if the exception is thrown – D-Dᴙum Nov 08 '22 at 20:19
  • Are you actually trying to find out the servers or the clients port? `HttpURLConnction` is only for client usage, but `BindException` usually occurs on the server side. – cyberbrain Nov 08 '22 at 20:44
  • I also thought about this discrepancy. But outgoing connections also need a port - usually picked from the `ephemeral port range`. If enough ports of that range are in use by (badly configured or highly active) other processes such errors can occur. – Queeg Nov 08 '22 at 20:48
  • @cyberbrain As a test, I created a Java program and ran it on the client to check all ports and return a status if busy or available. For the ones which were busy, a BindException was returned. I am assuming when Java makes the URL connection call, it attempts to open a local port (sort of like my test does) and it fails (similar to my test). – Unhandled Exception Nov 08 '22 at 20:56
  • @HiranChaudhuri Is that Java or the OS who selects the port to use? On one system, the range being used is actually outside the ephemeral port range and not sure why that is. I tried creating a post but its getting down voted so doubtful I will get any feedback on it. – Unhandled Exception Nov 08 '22 at 20:58
  • So to clarify: the client application has the problem that it fails to connect with a `BindException`? Or what exactly is the problem? Because with your actual question I would have to answer that you cannot log the port of the client, because the application on the client side doesn't specify that - independently of JVM or OS picking a client port. – cyberbrain Nov 09 '22 at 06:44
  • @cyberbrain The client application makes a number of calls to the server to retrieve various information. This works fine but then starts getting the BindException for every server call. After a about 15 seconds and numerous BindExceptions, those get cleared for some reason and the application returns to working normally. Eventually the BindExceptions will appear again, then go away again. I am trying to track down the client port(s) attempting to be used during these errors to see if the ports are not cycling/incrementing or are the ports being allocated actually in use. – Unhandled Exception Nov 09 '22 at 12:54

2 Answers2

0

UPDATE:

Your answers to the questions in the comments lead me to the thought that the application just opens more connections "at the same time" than allowed by the system. I quoted "at the same time" because as I wrote below, even if the application closes the connection, the port will be blocked by the TCP/IP stack for some time. I have to admit that I don't know whether the OS counts this as "used by the application" for that time.

Anyways your applications usually are not allowed to open all theoretically available ports at the same time.

If you use Linux or another Unix based OS (eg. MacOS), the number of open connections is counted against the open file descriptors. (On my Ubuntu system there is a limit of 1024 concurrently opened files.)

When you start the process from your terminal from bash, you can look at the ulimit command and find the limit with ulimit -a. For system-wide limitations, have a look at this SO answer to Increasing the maximum number of TCP/IP connections in Linux

For windows systems I did only a bit of search but cannot verify this, I found this article for Microsoft Windows 10: Troubleshoot port exhaustion issues

Original:

You will have to look at the stack trace of the exception to find out where it is triggered - maybe you can find the right class to enable logging for.

But it is not guaranteed that an application logs the network address it is trying to open.

When you describe that after some attempts it works again, you can try to find out with the command netstat what ports are actually opened, and also which processes use them. Look out for the state "TIME_WAIT": such connections are actively closed, but the networking stack still doesn't reuse the port as "old" packages from the recently opened connection could be on the way to the receiver. After a configured timeout, that port is then reusable again.

Find more details about this state in this SO question and its answers.

You also could try to attach a debugger to the active application, probably best with a built in decompiler or a plugin for that (usually IDEs like IntelliJ IDEA are able to do that) so you can set a breakpoint to the call of the method that throws the exception and inspect its parameters to find out about the port.

cyberbrain
  • 3,433
  • 1
  • 12
  • 22
  • Using netstat and it shows only a very very small number of established connections for the application on the client machine. The strange part is the port numbers used are outside the expected usage range. Not sure if that is somehow playing a part. – Unhandled Exception Nov 08 '22 at 20:29
  • So if you get the `BindException` on the HttpUrlConnection, it means you run out of free ports. What about connections for other applications? TCP/IP has "only" 65535 ports available per IP address so depending on the system at all and the application, it is not soo hard to run out of ports if you don't reuse an open connection for the same server, but open a lot of connections - even if you close them, it will need some time (as stated in my answer) until the port actually becomes free again, not only for the server side. – cyberbrain Nov 09 '22 at 13:40
  • Based on the netstat command there may only be 200-300 ports in use out of the available 16,000 based on the netsh output which shows the Start Port and Number of Ports value. – Unhandled Exception Nov 09 '22 at 13:50
  • @UnhandledException I updated my answer with a bit of information how to control the maximum number of concurrently opened ports (and files) – cyberbrain Nov 10 '22 at 13:15
  • Thanks for the update, the ulimit -a command is very useful on LInux but in this scenario it is Windows (is there a similar command for Windows, I couldn't find one.) . I created a simple server and a client app that loops and connects X number of times to the server. I set X to be in the thousands and it connects all within a few seconds. I have yet to run this program on the machine with the issue and curious to see if the results differ. – Unhandled Exception Nov 10 '22 at 18:24
  • I found this q/a for windows with a registry entry and links to further articles: https://learn.microsoft.com/en-us/answers/questions/495615/how-to-set-to-allow-maximumtcp-connection-in-windo.html – cyberbrain Nov 11 '22 at 07:40
0

In logging.properties you can increase log level. But if the programmer never placed a call to logging with the relevant data you will not see any relevant message in the logs anyway.

A safe way to figure out which socket the application is trying to open is to use strace.

Prefix your call to your application using strace, and maybe figure out which filters allow you to see network stuff only:

strace ... java ...

The first three dots need to be replaced with options for strace (and the filter for network traffic), the second three dots have to be replaced with everything your application needs to run.

strace will be a layer between an application (java) and the kernel, and it will print to console all system calls including the parameters and result codes. So you can see how the JVM is asking the OS to open a port and the OS responds with "nope, address already in use".

Queeg
  • 7,748
  • 1
  • 16
  • 42