0

I am copying data from in house Mysql database table to RDS MYSQL database table. i executed my AWS glue job on smaller data size ( 1million records) and i am able to successfully copy data to RDS Mysql database.

But when i try to run the same AWS glue job on table with 8 million records, it throws me below error

Failure Reason": "Communications link failure\n\nThe last packet successfully received from the server was 237,612 milliseconds ago. The last packet sent successfully to the server was 237,612 milliseconds ago."
user8545255
  • 761
  • 3
  • 9
  • 21
  • Have you tried the solutions listed here: https://stackoverflow.com/questions/2983248/com-mysql-jdbc-exceptions-jdbc4-communicationsexception-communications-link-fai – Leeroy Hannigan Dec 19 '22 at 19:31
  • Last packet was 237,612 milliseconds ago .. thats a bit ago, so if you copied this after that time frame, it never got the packets. Also, should be fine, but check your primary keys, sometimes when you COPY * from table SELECT * from sourcetable It works .. but it doesn't copy the primary keys .. have to set those again. Granted it sounds like that isn't the problem, but clearly some AWS setting is denying this, I know in normal MySQL there are literally hundreds of settings that can be adjusted that any one of them could deny access for many reasons (taking too long is one of them – easleyfixed Dec 19 '22 at 19:47
  • @easleyfixed i am able to copy the table with 3 million records using aws glue..but it throws me error with table that has 8 million records...no clue, what might be the root cause..trying to debug from last 2 days, but no luck... – user8545255 Dec 19 '22 at 20:01
  • As a test .. can you break the table in half and try doing a smaller chunk and see if that works? Find the actual limit .. is it the 4 million mark .. 5 ... etc... Then you can start trying to figure out WHY, is it a limit to rows in the table schema, is it upload time is past what it would expect so it errors out? Once you find out why you can try to increase some of the DB settings until you find out which one is halting the insert. I'm leaning towards row/character limit and perhaps timeout during the upload its self. – easleyfixed Dec 19 '22 at 20:03
  • @easleyfixed yup, i tested it by breaking into chunks.. No issues until 3 million records..it got copied in 11 min 15 sec..After 3 million, i just tried copying whole dataset 8 million records and it throws "communication link failure"... – user8545255 Dec 19 '22 at 20:10
  • Sounds like it is a timeout for a long connection. Let see if I can find something for you. – easleyfixed Dec 19 '22 at 20:15
  • Have a look here and see if any of this applies. https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-resolve-ssh-connection-errors/ and maybe over here https://stackoverflow.com/questions/2813843/possible-reasons-for-timeout-when-trying-to-access-ec2-instance – easleyfixed Dec 19 '22 at 20:17

0 Answers0