2

I have a MySQL database running inside an amazon t1.micro ec2 Linux instance.

I use a data source so I can do connection pooling, the code works fine on my home pc, but I get a communications link error when running the same exact code on the instance.

DataSource.java looks like this:

package resources;

import org.apache.commons.dbcp2.BasicDataSource;

public class DataSource {

    private static final String DRIVER_CLASS_NAME = "com.mysql.jdbc.Driver";
    private static final String DB_URL = "jdbc:mysql://ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com:3306/database";
    private static final String DB_USER = "ec2-user";
    private static final String DB_PASSWORD = "root";
    private static final int CONN_POOL_SIZE = 50;

    private BasicDataSource bds = new BasicDataSource();

    private DataSource() {
        bds.setDriverClassName(DRIVER_CLASS_NAME);
        bds.setUrl(DB_URL);
        bds.setUsername(DB_USER);
        bds.setPassword(DB_PASSWORD);
        bds.setInitialSize(CONN_POOL_SIZE);
    }

Then I have a jersey rest address which draws a connection like so:

source = DataSource.getInstance().getBds();
connection = source.getConnection();

When I run the project on my own pc, there's no errors and it fetches the data correctly. I can connect to the database remotely both via command line and from code with JDBC. But when I run/deploy the exact same code onto the amazon instance, I get this error trace:

java.sql.SQLException: Cannot create PoolableConnectionFactory (Communications link failure
Last packet sent to the server was 0 ms ago.)

[lower down]

Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Last packet sent to the server was 0 ms ago.

So apparently when ec2-user @ localhost tries to connect, the DB is not available.

the contents of my.cnf are:my.cnf

I have tried messing around with the bind address, setting it to 127.0.0.1 and commenting it out.

I am able to open mysql from linux command line with "mysql -u ec2-user -p root", and "mysql -h localhost -u ec2-user -p root". I confirmed that the server is in fact running on port 3306.

On my amazon instance security group i have the following inbound rules: https://i.stack.imgur.com/la0q2.png

I went through the steps in this post: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

  1. IP address or hostname in JDBC URL is wrong.

    • Cannot be as the same url works fine on another comp
  2. Hostname in JDBC URL is not recognized by local DNS server.

    • I tried /127.0.0.1:3306/database, /localhost:3306/database, /localhost/database, /ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com:3306/database, same error no matter what
  3. Port number is missing or wrong in JDBC URL.

    • it's 100% port 3306
  4. DB server is down.

    • not down as it accepts remote access
  5. DB server doesn't accept TCP/IP connections.

    • clearly it does accept if it works from my pc
  6. DB server has run out of connections. -||-

  7. Something in between Java and DB is blocking connections, e.g. a firewall or proxy.

    • the inbound rule is 0.0.0.0 so it should work form both local and remote

The user is "ec2-user"@"%" with all privileges granted, this user should work for both local and remote?

I am all out of things to try, most scenarios I came across are where people have it working locally but NOT remotely.

I am able to connect to localhost 3306 via telnet from the instance linux command line:

telnet localhost 3306 - Trying 127.0.0.1... Connected to localhost. Escape character is '^]'.
Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
marcs
  • 281
  • 1
  • 6
  • Can you check what is there in `/etc/hosts/` file? Also, can you try pinging website on public domain ex., www.amazon.com from the `amazon-ec2` host? – harshavmb May 26 '17 at 08:01
  • contents of /etc/hosts: 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost6 localhost6.localdomain6 – marcs May 26 '17 at 08:05
  • okay. Since, you are not trying to connect the mysql using localhost connection url, it could be something to do with dns. Can you try connecting mysql url `ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com:3306/database` from the `amazon-ec2` instance using mysql client? Don't use localhost, give the same url – harshavmb May 26 '17 at 08:07
  • ping www.amazon.com from the instance's linux command line gives: PING d3ag4hukkh62yn.cloudfront.net (13.32.206.248) 56(84) bytes of data. 64 bytes from 13.32.206.248: icmp_seq=1 ttl=241 time=9.59 ms 64 bytes from 13.32.206.248: icmp_seq=2 ttl=241 time=9.61 ms 64 bytes from 13.32.206.248: icmp_seq=3 ttl=241 time=9.56 ms --- d3ag4hukkh62yn.cloudfront.net ping statistics --- 18 packets transmitted, 18 received, 0% packet loss, time 144399ms rtt min/avg/max/mdev = 9.520/10.717/19.330/2.578 ms – marcs May 26 '17 at 08:09
  • okay. ping looks fine. not a firewall issue. Try to connect to mysqld having url `ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com:3306/databas‌​e` from the `amazon-ec2` instance. Could be something to do with DNS as there is nothing in `/etc/hosts/` file. – harshavmb May 26 '17 at 08:12
  • Also, did you comment `skip-networking` and `enable-named-pipe`if they exist in `my.cnf` file? – harshavmb May 26 '17 at 08:12
  • i also tried replacing the url with jdbc:mysql://localhost:3306/database but that yielded the same result. "mysql ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com:3306/databas‌​e" says "identifier name is too long" with just ip: mysql xx-xx-xx-xx:3306/database -p root "unknown database" is the syntax correct here? – marcs May 26 '17 at 08:15
  • "skip-networking" or "enable-named-pipe" aren't in my.cnf. – marcs May 26 '17 at 08:16
  • okay. While connecting using ip-address, you gave the public address of the machine or `127.0.0.1`? This appears more of a DNS lookup error to me. Can you run `nslookup ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com` ? – harshavmb May 26 '17 at 08:18
  • Hope you have `uncommented bind_address=0.0.0.0` to receive outbound requests. in your `my.cnf` I could see it commented. You shouldn't have connected to the mysql database, if it wasn't commented. Just wanted to know.. :) – harshavmb May 26 '17 at 08:22
  • nslookup ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com Server: [server] Address: [address] Non-authoritative answer: Name: ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com Address: [address] tryna be careful idk how much of this information is sensitive! :o – marcs May 26 '17 at 08:22
  • Didn't understand nslookup command output! – harshavmb May 26 '17 at 08:23
  • ok i uncommented bind-address 0.0.0.0 and restarted mysql, i thought leaving it commented was the same as allowing everything? – marcs May 26 '17 at 08:23
  • yeah.. everything – harshavmb May 26 '17 at 08:24
  • well, this doesn't seem to be DNS issue as well. Am running out of options here. From amazon ec2 host, when you tried to connect `ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com` to using mysql client, I think you got some error. When you replaced with the ip, you got a "unknown database" error. Can you check on that? Do you have database named `database` created in your mysql? – harshavmb May 26 '17 at 08:45
  • I'm 90% sure im entering the wrong command - is the literal thing that I enter into the command line "mysql ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com -u ec2-user -p" ? or how do u enter the command when using full url? also the database does exist yes – marcs May 26 '17 at 08:49
  • try `mysql -h ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com -u ec2-user -p` no need of double quotes. No sooner you hit enter, password prompt will be shown, enter the password – harshavmb May 26 '17 at 08:51
  • yeah, using that command it does connect from the instance command line. – marcs May 26 '17 at 08:53
  • okies, can you run your application and test? Both, locally and remotely – harshavmb May 26 '17 at 08:54
  • ran from netbeans 8.2 using glassfish 4.1, works as intended. refreshed the app @ ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com, same error. Could the server have anything to do with it? I'm using deployment straight from netbeans to a a preconfigured amazon glassfish beanstalk – marcs May 26 '17 at 09:02
  • If you are able to connect from mysql client, then you should be able to connect. Not sure about conflict with glassfish. Is Amazon glassfish hosted on the same machine or on a different host? – harshavmb May 26 '17 at 09:09
  • If i understand correctly, when you create an environment for an elastic beanstalk application, it creates (in my case) an ec2-instance with some kind of a docker on top that you can put your already finished application on. for me it says "64bit Debian jessie v2.6.0 running GlassFish 4.1 Java 8 (Preconfigured - Docker)". AFAIK this is just an easier way of having jvm and glassfish ready to go so you dont have to set them up yourself. I just connected to the ec2-instance that was created and installed mysql server on it. – marcs May 26 '17 at 09:15
  • hmm., Do we need to specify these db configurations else where in glassfish as well? – harshavmb May 26 '17 at 09:23
  • is it supposed to need some kind of configuration for glassfish? The mysql server is running kind of as a separate entity and the connection is handled by JDBC, right? I didn't set any configs anywhere, I assumed it would be the same as when i run it from netbeans where it also uses glassfish 4.1 and is able to connect to a mysql server running on my pc that it has no knowledge of other than the url/username/password given to the datasource – marcs May 26 '17 at 09:41
  • Usually, app servers will also hold datasource specifications, I remember configuring them on websphere and jboss serverz. Unsure about glassfish. Are you seeing any errors/warnings/alerts in mysql database logs? – harshavmb May 26 '17 at 09:52
  • Contents of /var/log/mysqld.log: http://i.imgur.com/knw8H4Q.png are there other mysql logs that might contain relevant info? – marcs May 26 '17 at 16:45
  • Okay. Am not seeing useful logs there. Can you try connecting from MySQL client and from your app. See if you see any difference in MySQL logs... – harshavmb May 26 '17 at 16:50
  • the log only updates when i restart mysql, when i connect from command line or the app there's no new lines added after "ready for connections. Version '5.6.35' socket '/var/lib/mysql/mysql.sock' port:3308 MySQL Community server (GPL)" .Is this supposed to be so? – marcs May 26 '17 at 17:05
  • Actually errors of mysqld process are captured in mysqld.local.err file. I'm using Mac. May be a different file in your *nix flavour but inside data directory. – harshavmb May 26 '17 at 17:24
  • I ran "sudo ls -l `pgrep mysqld | sed 's@.*@/proc/&/fd/@'` | grep log" to see where it logs, and the result was: l-wx------ 1 root root 64 May 26 17:38 1 -> /var/log/mysqld.log l-wx------ 1 root root 64 May 26 17:38 2 -> /var/log/mysqld.log lrwx------ 1 root root 64 May 26 17:38 8 -> /var/lib/mysql/ib_logfile0 lrwx------ 1 root root 64 May 26 17:38 9 -> /var/lib/mysql/ib_logfile1 . The ib_logfiles are just binary, so i guess it only logs to mysqld.log? – marcs May 26 '17 at 17:38
  • Yeah ib_logfiles are innodb engine binary files. Mysqld.log should log them. Sorry, am not aware of this issue. Looks like a difficult one to identify. Have you considered raising this with Amazon support? They might help you. I guess we do have community support which is free.. – harshavmb May 26 '17 at 17:42
  • yes it's a very strange issue. I'll contant amazon support – marcs May 26 '17 at 17:44
  • Good luck! Please do update huh! – harshavmb May 26 '17 at 17:45

2 Answers2

0

Issue avoided by simply running the mysql database on a separate ec2- instance. I guess there's no answer to the original setup but... this works right now.

marcs
  • 281
  • 1
  • 6