5

In my application which runs on Jboss server, we make an http connection to another application. Both the application lives in the same datacenter. We use the VIP to connect to the application. This works fine in production except for a very small percentage when we get "java.net.UnknownHostException". The percentage is very low (~0.2%), but still considering the huge volume of requests we get, the actual number cannot be ignored.

The error is intermittent and does not follow a regular pattern neither in terms of time, nor frequency. Sometimes we get this 200 times in a day and sometimes we do not get it for 3-4 consecutive days. Sometimes we even get it when the expected traffic is low (1-3 AM at night).

AFAIK, this particular exception comes when the DNS is not able to resolve the IP for a given hostname. We looked into the DNS settings/configurations etc and everything looks good. Do not want to use the IP address directly in the properties file or in /etc/hosts file.

At this point, I am not sure what I can do next to debug further. Any help here/guidance would be much appreciated.

    Caused by: java.net.UnknownHostException: <VIP Name>
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.6.0_25]
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:850) ~[?:1.6.0_25]
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1201) ~[?:1.6.0_25]
at java.net.InetAddress.getAllByName0(InetAddress.java:1154) ~[?:1.6.0_25]
at java.net.InetAddress.getAllByName(InetAddress.java:1084) ~[?:1.6.0_25]
at java.net.InetAddress.getAllByName(InetAddress.java:1020) ~[?:1.6.0_25]
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:242) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:130) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:573) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) ~[httpclient-4.1.2.jar:4.1.2]
at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:88) ~[spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:46) ~[spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:49) ~[spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:438) ~[spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:401) ~[spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]
user1270392
  • 2,981
  • 4
  • 21
  • 25
  • 1
    Have you considered doing a non-Java test of the DNS that the hosts use? To me it seems unlikely this is a Java problem. – Philip Whitehouse Mar 01 '13 at 20:05
  • the most basic thing you can do is to deploy a monitoring tool like [Smokeping](http://oss.oetiker.ch/smokeping/) to see what's going on with your DNS server. Apart from ping probes, Smokeping generates nice graphs from periodic DNS probes. [Here's an example](http://oss.oetiker.ch/smokeping-demo/?target=rootns) – dschulz Mar 01 '13 at 20:17
  • Thanks for the reply. To me also it doesn't seem to be a Java problem. I agree that we can deploy a monitoring tool which can provide the full network traffic going in and out of the server. However, I am planning to start like this. Though not sure whether it makes complete sense or not :) - Adding the IP-Host combination in /etc/hosts file temporarily. Watching it for next few hrs. If the error goes away, then for sure its related only to DNS. Then I will proceed with debugging with monitoring tool. – user1270392 Mar 01 '13 at 20:29
  • One more thing that comes to my mind. I know that on these systems the ulimit is not optimally set. The question is that can having a low ulimit cause the UnknownHostException? or it will throw another exception if the ulimit has increased? – user1270392 Mar 01 '13 at 22:41
  • 1
    @user1270392 Just out of curiosity, I don't suppose you ever found the root cause of this? I'm experiencing something similar, and also don't think Java is to blame, but if you found it was for some reason then I'd be interested to know. – Michael Berry Jan 26 '18 at 17:55
  • I'm seeing this too on Linux 64/JDK 12. I can say "ping tinybrain.de" in the console, and it works. Then I start a Java program that shows UnknownHostException with that same host. Extremely weird. – Stefan Reich Apr 14 '19 at 17:55

0 Answers0