Oracle Java in Docker Container cannot resolve hostname after /etc/nsswitch.conf changed

Question

I was minimizing Oracle-JRE 8 Docker Image to deploy my own playframework app.

Host: CentOS 7 minimal + Oracle JRE 8

The steps are as following:

Create bash docker image with busybox, since playframework generate bash based startup scripts. I built bash binaries, and use ldd /bin/bash to find related lib so files, and ADD them into the image.
Create Oracle JRE 8 image from bash-with-busybox with libm-2.17.so libm.so.6 libpthread-2.17.so libpthread.so.0
Create application container image with some startup scrips from above
Create playframework container image from above

When I start up the playframework application I came across the following error, which is

Caused by: java.net.UnknownHostException: mysql.server

But when I tried ping mysql.server telnet mysql.server 3306 both are working.

And then I searched from internet, and I found some article mentioned that Oracle Java requires nss to resovle server(domain) names, the following command was suggested

echo 'networkaddress.cache.ttl=10' >> $JAVA_HOME/jre/lib/security/java.security

echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf

I tried, but the hostname still cannot be resolved.

Since I used docker network create a private network, so the /etc/hosts file is as following:

  172.18.0.6      4de243ab0668
  127.0.0.1       localhost
  ::1     localhost ip6-localhost ip6-loopback
  fe00::0 ip6-localnet
  ff00::0 ip6-mcastprefix
  ff02::1 ip6-allnodes
  ff02::2 ip6-allrouters
  172.18.0.4      redis.server.imadz
  172.18.0.5      mysql.server
  172.18.0.5      mysql.server.imadz
  172.18.0.6      rest.server.imadz
  172.18.0.2      activemq.server
  172.18.0.2      activemq.server.imadz
  172.18.0.4      redis.server
  172.18.0.3      mongo.server
  172.18.0.3      mongo.server.imadz
  172.18.0.6      rest.server

The related exception is as following:

bash-4.3# Java HotSpot(TM) 64-Bit Server VM warning: No monotonic clock was available - timed services may be adversely affected if the time-of-day clock changes
Picked up JAVA_TOOL_OPTIONS: -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9999
Configuration error: Configuration error[Cannot connect to database [db-server-name]]
at play.api.Configuration$.play$api$Configuration$$configError(Configuration.scala:94)
at play.api.Configuration.reportError(Configuration.scala:743)
at play.api.db.BoneCPPlugin$$anonfun$onStart$1.apply(DB.scala:247)
at play.api.db.BoneCPPlugin$$anonfun$onStart$1.apply(DB.scala:238)
at scala.collection.immutable.List.map(List.scala:273)
at play.api.db.BoneCPPlugin.onStart(DB.scala:238)
at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:91)
at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:91)
at scala.collection.immutable.List.foreach(List.scala:381)
at play.api.Play$$anonfun$start$1.apply$mcV$sp(Play.scala:91)
at play.api.Play$$anonfun$start$1.apply(Play.scala:91)
at play.api.Play$$anonfun$start$1.apply(Play.scala:91)
at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
at play.api.Play$.start(Play.scala:90)
at play.core.StaticApplication.<init>(ApplicationProvider.scala:55)
at play.core.server.NettyServer$.createServer(NettyServer.scala:244)
at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:280)
at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:275)
at scala.Option.map(Option.scala:146)
at play.core.server.NettyServer$.main(NettyServer.scala:275)
at play.core.server.NettyServer.main(NettyServer.scala)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:356)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2504)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2541)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2323)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:363)
at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:120)
at play.api.db.BoneCPPlugin$$anonfun$onStart$1.apply(DB.scala:240)
... 18 more
Caused by: java.net.UnknownHostException: mysql.server
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:250)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:306)
... 36 more

I got no clue at all, is it because some required linux libraries are needed or certain binary are required? or it is some other problem, what should be the direction to figure out this issue?

mini-oraclejre8 Dockerfile

FROM bash-on-busybox

RUN mkdir -p /usr/local/lib
ADD server-jre-8u65-linux-x64.tar.gz /usr/local/lib
ADD jre.tar.gz /lib64
ENV JAVA_HOME /usr/local/lib/jdk1.8.0_65
ENV PATH      $JAVA_HOME/bin:$PATH

# Fix DNS resolution issues when nss is not installed
RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >>     /etc/nsswitch.conf

# Set DNS cache to 10 seconds (Cache is permanent by default). Network hosts are volatile in Docker clusters.
RUN grep '^networkaddress.cache.ttl=' $JAVA_HOME/jre/lib/security/java.security || echo 'networkaddress.cache.ttl=10' >> /$JAVA_HOME/jre/lib/security/java.security

so files inside lib64

  drwxr-xr-x    2 root     root          4096 Dec 30 09:39 .
  drwxr-xr-x   13 root     root          4096 Jan  1 12:36 ..
  -rwxr-xr-x    1 root     root        164440 Nov 30 20:31 ld-2.17.so
  lrwxrwxrwx    1 root     root            10 Dec 28 14:37 ld-linux-x86-64.so.2 -> ld-2.17.so
  -rwxr-xr-x    1 root     root       2107816 Nov 30 20:31 libc-2.17.so
  lrwxrwxrwx    1 root     root            12 Dec 28 14:37 libc.so.6 -> libc-2.17.so
  -rwxr-xr-x    1 root     root         19520 Nov 30 20:31 libdl-2.17.so
  lrwxrwxrwx    1 root     root            13 Dec 28 14:37 libdl.so.2 -> libdl-2.17.so
  -rwxr-xr-x    1 root     root       1141560 Nov 30 20:31 libm-2.17.so
  lrwxrwxrwx    1 root     root            12 Dec 28 14:37 libm.so.6 -> libm-2.17.so
  -rwxr-xr-x    1 root     root        142304 Nov 30 20:31 libpthread-2.17.so
  lrwxrwxrwx    1 root     root            18 Dec 28 14:37 libpthread.so.0 -> libpthread-2.17.so
  lrwxrwxrwx    1 root     root            15 Dec 28 13:07 libtinfo.so.5 -> libtinfo.so.5.9
  -rwxr-xr-x    1 root     root        174520 Jun 10  2014 libtinfo.so.5.9

Could you perhaps provide a sample Dockerfile so that someone could reproduce your problem? As it stands all we have is your analysis. Are you certain this is not a docker networking problem? You appear to be assuming it's a problem with Oracle JRE. If that is the case how is it related to Docker? — Mark O'Connor, Jan 01 '16 at 11:53
Because inside Container, I can use telnet mysql.server 3306 or ping mysql.server to test the connection, and both of them works. I do not know what is under the cover of java networking api (InetAddress, and NameService) — Barry Zhong, Jan 01 '16 at 12:22
So you've ruled out Docker networking. Why not eliminate Docker completely and recreate the problem with a standalone Java process? — Mark O'Connor, Jan 01 '16 at 12:33
This is most likely to be a problem with the ACL settings in mysql. Pinging the port is not enough you need to actually connect to the DB using a mysql client. See: http://stackoverflow.com/questions/2983248/com-mysql-jdbc-exceptions-jdbc4-communicationsexception-communications-link-fai and http://stackoverflow.com/questions/6865538/solving-a-communications-link-failure-with-jdbc-and-mysql — Mark O'Connor, Jan 01 '16 at 12:37
Thanks for replies, those are not the same question, those are ACL related with exception Caused by: java.net.ConnectException: Connection refused, but mine is different, it is java.net.UnknownHostException: mysql.server — Barry Zhong, Jan 01 '16 at 12:41

score 0 · Accepted Answer · answered Jan 02 '16 at 10:34

After I added more share libraries it works.

      [root@localhost eb-rest]# docker exec b82c214c6375 ls -al /lib64
  total 4676
  drwxr-xr-x    2 root     root          4096 Jan  1 16:36 .
  drwxr-xr-x   14 root     root          4096 Jan  1 17:08 ..
  -rwxr-xr-x    1 root     root        164440 Nov 30 20:31 ld-2.17.so
  lrwxrwxrwx    1 root     root            10 Dec 28 14:37 ld-linux-x86-64.so.2 -> ld-2.17.so
  -rwxr-xr-x    1 root     root          8568 Nov 30 20:31 libBrokenLocale-2.17.so
  lrwxrwxrwx    1 root     root            32 Dec 30 03:50 libBrokenLocale.so -> ../../lib64/libBrokenLocale.so.1
  lrwxrwxrwx    1 root     root            23 Dec 28 14:37 libBrokenLocale.so.1 -> libBrokenLocale-2.17.so
  -rwxr-xr-x    1 root     root         22048 Nov 30 20:31 libSegFault.so
  -rwxr-xr-x    1 root     root         19944 Nov 30 20:31 libanl-2.17.so
  lrwxrwxrwx    1 root     root            23 Dec 30 03:50 libanl.so -> ../../lib64/libanl.so.1
  lrwxrwxrwx    1 root     root            14 Dec 28 14:37 libanl.so.1 -> libanl-2.17.so
  -rwxr-xr-x    1 root     root       2107816 Nov 30 20:31 libc-2.17.so
  -rw-r--r--    1 root     root           253 Nov 30 20:00 libc.so
  lrwxrwxrwx    1 root     root            12 Dec 28 14:37 libc.so.6 -> libc-2.17.so
  -rwxr-xr-x    1 root     root        201160 Nov 30 20:31 libcidn-2.17.so
  lrwxrwxrwx    1 root     root            24 Dec 30 03:50 libcidn.so -> ../../lib64/libcidn.so.1
  lrwxrwxrwx    1 root     root            15 Dec 28 14:37 libcidn.so.1 -> libcidn-2.17.so
  -rwxr-xr-x    1 root     root         40816 Nov 30 20:31 libcrypt-2.17.so
  lrwxrwxrwx    1 root     root            25 Dec 30 03:50 libcrypt.so -> ../../lib64/libcrypt.so.1
  lrwxrwxrwx    1 root     root            16 Dec 28 14:37 libcrypt.so.1 -> libcrypt-2.17.so
  -rwxr-xr-x    1 root     root         19520 Nov 30 20:31 libdl-2.17.so
  lrwxrwxrwx    1 root     root            22 Dec 30 03:50 libdl.so -> ../../lib64/libdl.so.2
  lrwxrwxrwx    1 root     root            13 Dec 28 14:37 libdl.so.2 -> libdl-2.17.so
  -rwxr-xr-x    1 root     root       1141560 Nov 30 20:31 libm-2.17.so
  lrwxrwxrwx    1 root     root            21 Dec 30 03:50 libm.so -> ../../lib64/libm.so.6
  lrwxrwxrwx    1 root     root            12 Dec 28 14:37 libm.so.6 -> libm-2.17.so
  -rwxr-xr-x    1 root     root         23440 Nov 30 20:31 libmemusage.so
  -rwxr-xr-x    1 root     root        113328 Nov 30 20:31 libnsl-2.17.so
  lrwxrwxrwx    1 root     root            23 Dec 30 03:50 libnsl.so -> ../../lib64/libnsl.so.1
  lrwxrwxrwx    1 root     root            14 Dec 28 14:37 libnsl.so.1 -> libnsl-2.17.so
  -rwxr-xr-x    1 root     root         46552 Nov 30 20:31 libnss_compat-2.17.so
  lrwxrwxrwx    1 root     root            30 Dec 30 03:50 libnss_compat.so -> ../../lib64/libnss_compat.so.2
  lrwxrwxrwx    1 root     root            21 Dec 28 14:37 libnss_compat.so.2 -> libnss_compat-2.17.so
  -rwxr-xr-x    1 root     root         38224 Nov 30 20:31 libnss_db-2.17.so
  lrwxrwxrwx    1 root     root            26 Dec 30 03:50 libnss_db.so -> ../../lib64/libnss_db.so.2
  lrwxrwxrwx    1 root     root            17 Dec 28 14:37 libnss_db.so.2 -> libnss_db-2.17.so
  -rwxr-xr-x    1 root     root         27512 Nov 30 20:31 libnss_dns-2.17.so
  lrwxrwxrwx    1 root     root            27 Dec 30 03:50 libnss_dns.so -> ../../lib64/libnss_dns.so.2
  lrwxrwxrwx    1 root     root            18 Dec 28 14:37 libnss_dns.so.2 -> libnss_dns-2.17.so
  -rwxr-xr-x    1 root     root         61928 Nov 30 20:31 libnss_files-2.17.so
  lrwxrwxrwx    1 root     root            29 Dec 30 03:50 libnss_files.so -> ../../lib64/libnss_files.so.2
  lrwxrwxrwx    1 root     root            20 Dec 28 14:37 libnss_files.so.2 -> libnss_files-2.17.so
  -rwxr-xr-x    1 root     root         28264 Nov 30 20:31 libnss_hesiod-2.17.so
  lrwxrwxrwx    1 root     root            30 Dec 30 03:50 libnss_hesiod.so -> ../../lib64/libnss_hesiod.so.2
  lrwxrwxrwx    1 root     root            21 Dec 28 14:37 libnss_hesiod.so.2 -> libnss_hesiod-2.17.so
  -rwxr-xr-x    1 root     root         56792 Nov 30 20:31 libnss_nis-2.17.so
  lrwxrwxrwx    1 root     root            27 Dec 30 03:50 libnss_nis.so -> ../../lib64/libnss_nis.so.2
  lrwxrwxrwx    1 root     root            18 Dec 28 14:37 libnss_nis.so.2 -> libnss_nis-2.17.so
  -rwxr-xr-x    1 root     root         65744 Nov 30 20:31 libnss_nisplus-2.17.so
  lrwxrwxrwx    1 root     root            31 Dec 30 03:50 libnss_nisplus.so -> ../../lib64/libnss_nisplus.so.2
  lrwxrwxrwx    1 root     root            22 Dec 28 14:37 libnss_nisplus.so.2 -> libnss_nisplus-2.17.so
  -rwxr-xr-x    1 root     root          8888 Nov 30 20:31 libpcprofile.so
  -rwxr-xr-x    1 root     root        142304 Nov 30 20:31 libpthread-2.17.so
  -rw-r--r--    1 root     root           222 Nov 30 20:01 libpthread.so
  lrwxrwxrwx    1 root     root            18 Dec 28 14:37 libpthread.so.0 -> libpthread-2.17.so
  -rwxr-xr-x    1 root     root        110808 Nov 30 20:31 libresolv-2.17.so
  lrwxrwxrwx    1 root     root            26 Dec 30 03:50 libresolv.so -> ../../lib64/libresolv.so.2
  lrwxrwxrwx    1 root     root            17 Dec 28 14:37 libresolv.so.2 -> libresolv-2.17.so
  -rwxr-xr-x    1 root     root         44096 Nov 30 20:31 librt-2.17.so
  lrwxrwxrwx    1 root     root            22 Dec 30 03:50 librt.so -> ../../lib64/librt.so.1
  lrwxrwxrwx    1 root     root            13 Dec 28 14:37 librt.so.1 -> librt-2.17.so
  -rwxr-xr-x    1 root     root         38352 Nov 30 20:31 libthread_db-1.0.so
  lrwxrwxrwx    1 root     root            29 Dec 30 03:50 libthread_db.so -> ../../lib64/libthread_db.so.1
  lrwxrwxrwx    1 root     root            19 Dec 28 14:37 libthread_db.so.1 -> libthread_db-1.0.so
  lrwxrwxrwx    1 root     root            15 Dec 28 13:07 libtinfo.so.5 -> libtinfo.so.5.9
  -rwxr-xr-x    1 root     root        174520 Jun 10  2014 libtinfo.so.5.9
  -rwxr-xr-x    1 root     root         14608 Nov 30 20:31 libutil-2.17.so
  lrwxrwxrwx    1 root     root            24 Dec 30 03:50 libutil.so -> ../../lib64/libutil.so.1
  lrwxrwxrwx    1 root     root            15 Dec 28 14:37 libutil.so.1 -> libutil-2.17.so

how did you go about determining these were the missing libraries? — Mark Elliot, Mar 18 '19 at 22:32
Compared with those container working. I am not sure those were the minimal set. — Barry Zhong, Mar 24 '19 at 06:42

Oracle Java in Docker Container cannot resolve hostname after /etc/nsswitch.conf changed

1 Answers1