Unable to access tomcat manager 8080 in Google Cloud

Question

I have been using Amazon EC2 to run my Tomcat+MySQL website for a while and is now migrating to Google Cloud Platform. I start a compute engine instance (Ubuntu 16.04), connect to it via ssh and use apt-get to install mysql/tomcat7.

The problem I encountered is that tomcat will not start. The catalina.out log didn't have a "Server startup at xxxms" message, and I can't connect to 8080 port via browser.

The last several lines of catalina.out is

Jul 10, 2017 7:06:20 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 928 ms
Jul 10, 2017 7:06:20 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina
Jul 10, 2017 7:06:20 PM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.68 (Ubuntu)
Jul 10, 2017 7:06:20 PM org.apache.catalina.startup.HostConfig deployDescriptor INFO: Deploying configuration descriptor /etc/tomcat7/Catalina/localhost/host-manager.xml
Jul 10, 2017 7:06:21 PM org.apache.catalina.startup.TldConfig execute INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

When I use netstat to check, it shows user tomcat7 is listening to 8080

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      115        32984       -   

$ id -u tomcat7  
$ 115           

I try to wget localhost:8080 in ssh terminal, it shows

Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 

and just hang there.

Any idea or suggestion will be greatly appreciated!

Update

It turns out that firewall is not the root cause of the problem, and even without allowing port 8443 Tomcat will work (Of coz you need to allow 8080). The reason that there's no "Server started" message showing up is Tomcat take extremely long time to startup (1346049 ms the first time, 354034 ms when restarted, no web app installed except for the default index.html), and the reason for no responding to request is also that it has not finished starting up yet.

This is the first time I have seen that Tomcat takes so long to start and also the reason I didn't realize it in the first place. I suspect (with some search) this is caused by Tomcat Jar scanning. Will keep update this question once I have more detail.

Update - Problem Solved

It turns out that I encounter the same problem here and the solution is here. In short, much of the time is consumed by the following task:

Creation of SecureRandom instance for session ID generation using [SHA1PRNG]

which require Java to load /dev/random to get random numbers. /dev/random typically get its entropy source from keyboard/mouse input, which cannot provide enough randomness on a headless virtual machine. This causes the random number to be "used up" during computation and cause a lot of wait. The solution is to install haveged, which use some other source to provide randomness (details in the link).

I installed haveged, and now tomcat only takes 1 sec to startup and everything works normal.

Answer 1

Thanks for asking such interesting question.

I've never used Google Cloud services but I managed to replicate your issue.

After reading a little I found that you need to update your Firewall Rules to enable access to 8080 port.

Go to:

1) (Hamburguer Icon, upper left) 2) Networking 3) Firewall Rules 4) Add new

I created one called 'allow-tomcat7' with this properties:

Descripción

Enables Tomcat 7 access
Red

default
Prioridad

1000
Dirección

Entrada
Acción tras coincidencia

Permitir
Filtros de origen

Intervalos de IP    
0.0.0.0/0
Protocolos y puertos

tcp:8080
tcp:8443
udp:8080

There's an option for 'target tags' when you edit the configuration, although I've created a 'tag' and applied it only to my new EC instance it didn't work. I had to remove target tags and it worked like a charm:

Make sure you allow access only for your IP address!

You'll need to adjust your security settings, otherwise, you'll become a honeypot, once I've enabled the port for everyone several bots started to scan it:

daychuzleo@testing-tomcat:~$ sudo tcpdump -i ens4 port 8080
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), capture size 262144 bytes

20:39:31.437634 IP 170.251.221.183.54162 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [.], seq 1638030511:1638030512, ack 1250919796, win 259, length 1: HTTP
20:39:31.437665 IP testing-tomcat.c.hip-river-163201.internal.http-alt > 170.251.221.183.54162: Flags [.], ack 1, win 231, options [nop,nop,sack 1 {0:1}], length 0

20:39:37.133899 IP 170.251.221.183.53878 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [.], seq 2436191518:2436191519, ack 4071767590, win 259, length 1: HTTP
20:39:37.133930 IP testing-tomcat.c.hip-river-163201.internal.http-alt > 170.251.221.183.53878: Flags [.], ack 1, win 222, options [nop,nop,sack 1 {0:1}], length 0

20:39:51.379839 IP 170.251.221.183.54162 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [F.], seq 1, ack 1, win 259, length 0
20:39:51.392375 IP 170.251.221.183.47923 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [S], seq 1420913913, win 8192, options [mss 1386,nop,wscale 8,nop,nop,sackOK,unknown-76 0x01010a18e9680005,unknown-76 0x0c01,nop,eol], length 0
20:39:51.392410 IP testing-tomcat.c.hip-river-163201.internal.http-alt > 170.251.221.183.47923: Flags [S.], seq 507557961, ack 1420913914, win 28400, options [mss 1420,nop,nop,sackOK,nop,wscale 7], length 0
20:39:51.421934 IP testing-tomcat.c.hip-river-163201.internal.http-alt > 170.251.221.183.54162: Flags [.], ack 2, win 231, length 0
20:39:51.586555 IP 170.251.221.183.47923 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [.], ack 1, win 259, length 0
20:39:51.590317 IP 170.251.221.183.47923 > testing-tomcat.c.hip-river-163201.internal.http-alt: Flags [P.], seq 1:389, ack 1, win 259, length 388: HTTP: GET / HTTP/1.1
20:39:51.590337 IP testing-tomcat.c.hip-river-163201.internal.http-alt > 170.251.221.183.47923: Flags [.], ack 389, win 231, length 0

I was unable to make it work with wget, but I think with this you'll found it out.

UPDATE:

I forgot to mention some things you may have not configured:

-Allowing Firewall for HTTP and HTTPS in your VM instance:

• Try using a web navigator (Chrome, Firefox) don't use wget.

• Verify that you're not being filtered by your company firewall, try testing with 4g in your cell phone or an unrestricted network, or just ask your IT team to allow you access to the temporary public IP (and port) generated (each time).

  • Start the service using:

  sudo service tomcat7 start

  • Try reinstalling tomcat

Other things I did (in the research process)

• Moving the service to IPV4 instead of IPV6

  daychuzleo@testing-tomcat:~$ netstat -ntpl (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
  tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN -
  tcp6 0 0 :::22 :::* LISTEN -

To do it, edit the default tomcat and add in JavaOPTS the IPV4 option:

vim /etc/default/tomcat

JAVA_OPTS="-Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true"

Disable the 8443 redirection

Comment the section "redirectPort" in server.xml:

vim /etc/tomcat/server.xml 

 <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               address="0.0.0.0"/>
               <!--redirectPort="8443" />-->

Verify each change by restarting your tomcat instance.