22

I am still starting out with AWS Glue and I am trying to connect it to my publicly accessible MySql database hosted on RDS Aurora to get its data.

So I start by creating a crawler and in the data store I create a new connection as in the screenshot below: enter image description here

I go through the rest and eventually try to run the crawler but I get the following error: At least one security group must open all ingress ports.To limit traffic, the source security group in your inbound rule can be restricted to the same security group

I am not sure what I need to change in the security group attached to the RDS but here's what I have right now for the inbound rules:

enter image description here

You'll notice that I have a self-referencing rule in there that's pointing to the same security group.

The outbound rules are going to all traffic.

Any idea what I might be doing wrong?

Vzzarr
  • 4,600
  • 2
  • 43
  • 80
Naguib Ihab
  • 4,259
  • 7
  • 44
  • 80

4 Answers4

21

The inbound rule (Glue Connection security group) is set to allow TCP Port 0 to allow traffic. Instead, it should allow ALL traffic. Edit your rules, and where there's a dropdown that says "Custom TCP Rule", and change it to "All TCP".

The documentation explains how to setup the security group

Abdelrahman Maharek
  • 802
  • 5
  • 19
Christopher Armstrong
  • 2,107
  • 2
  • 14
  • 22
  • 2
    Thanks, I am still getting an error though but I think that's more related to the output of the crawler: `VPC S3 endpoint validation failed for SubnetId: subnet-1944ab40. VPC: vpc-c8605bad. Reason: Could not find S3 endpoint or NAT gateway for subnetId: subnet-1944ab40 in Vpc vpc-c8605bad` – Naguib Ihab Aug 13 '18 at 03:58
  • 1
    Yeah, that's a different problem, you may want to create a separate question. I believe it's relating to the need to set up what's called an "S3 Gateway Endpoint" in your VPC / subnet. – Christopher Armstrong Aug 27 '18 at 20:49
  • 1
    Did the second issue ever get resolved? Experiencing the same issue – openwonk May 02 '19 at 03:37
6

You need to set a new rule in the security group that is attached to your DB instances where you define:

  • Type: All TCP
  • Protocol: TCP
  • Range: 0 - 65535
  • Source: Custom sg-(the id of this/self security group)
  • Description: whatever you want
Vzzarr
  • 4,600
  • 2
  • 43
  • 80
David I. Rock
  • 95
  • 2
  • 4
3

To solve the second error mentioned above in the comments (VPC S3 endpoint validation failed for SubnetId: subnet-1944ab40. VPC: vpc-c8605bad. Reason: Could not find S3 endpoint or NAT gateway for subnetId: subnet-1944ab40 in Vpc vpc-c8605bad) you have to create an Amazon VPC Endpoints for Amazon S3. https://docs.aws.amazon.com/glue/latest/dg/vpc-endpoints-s3.html

jlwhite
  • 31
  • 1
0

I found @David I. Rock solution to be working but has the inconvenience to stop connections via SQL Clients.

On top of that I also added the inbound rule:

  • Type: MYSQL / Aurora
  • Protocol: TCP (automatically generated)
  • Port Range: 3306 (automatically generated)
  • Source: My IP (or adapt to your requirements)
Vzzarr
  • 4,600
  • 2
  • 43
  • 80