Stack Overflow
Stack Overflow

Questions tagged [aws-glue-connection]

40 questions
4
votes
2 answers

Cross-region AWS Glue connection

What do I have: AWS Glue connection in region A, RDS MySQL database in region B. I want to configure access from AWS Glue to the database. I have peering between two regions. Security group is used for AWS Glue connection meets the VPC requirements.…
Tarasovych
  • 2,228
  • 3
  • 19
  • 51
4
votes
0 answers

AWS GLUE: MySQL through SSH Tunnel

I need to connect to an on-prem database that I normally access from within the enterprise VPN, and then through an SSH tunnel. We already have a VPC on AWS configured for our own VPN, but I could not figure out how to create a Glue Connection with…
kazantzakis
  • 73
  • 4
4
votes
1 answer

AWS GlueStudio to Snowflake JDBC: An error occurred while calling pyWriteDynamicFrame. No suitable driver

I'm trying to move data from a Data Catalog table (MySQL) through AWS Glue (in visual mode of GlueStudio) into a snowflake table. For this, I'm following this guide Performing data transformations using Snowflake and AWS Glue [1] I'm following every…
Daniel Salcedo
  • 322
  • 3
  • 15
4
votes
3 answers

AWS Glue JDBC connection created with CDK needs password in the console before it becomes valid

I am creating a JDBC connection in Glue using secrets for username and password. I can see in the console that username is read correctly from the secret, so that's not a concern. Once I edit the details and enter the password in the console, it…
Krzysztof Słowiński
  • 6,239
  • 8
  • 44
  • 62
2
votes
0 answers

Glue secret manager integration: secretId is not provided

I am running the glue pyspark script from my local machine using the GlueETL library. When creating a dataframe from glue catalog, dyf_user_book_reading_stat = glueContext.create_dynamic_frame.from_catalog( database="xxx-db", …
sheetal_158
  • 7,391
  • 6
  • 27
  • 44
2
votes
1 answer

Permission bigquery.tables.get denied or it may not exist

I am using the AWS Glue connector for BigQuery. My glue jobs were running fine in multiple AWS accounts but suddenly it started failing with the below response in all the accounts together: Access Denied: Table…
cpdevra
  • 21
  • 1
  • 3
1
vote
1 answer

AWS Glue Crawler and JDCBConnection : "Expected string length >= 1, but found 0 for params.Targets.JdbcTargets[0].customJdbcDriverClassName"

I am trying to setup an AWS Glue Crawler using a JDBC connection in order to populate my AWS Glue Data Catalog databases. I already have a Connection which passes the test but when I submit my crawler creation, I have this error : "Expected string…
Thomas Heniart
  • 23
  • 3
1
vote
0 answers

AWS Glue connection failing with self-hosted MongoDB on EC2

I'm trying to establish an AWS Glue connection with MongoDB on self-hosted EC2. The connection is in the same subnet(private) as MongoDB. The connection has same security group as the MongoDB instance, with all the ports open within the same SG. No…
Omisha gupta
  • 121
  • 1
  • 6
1
vote
1 answer

Using custom connector in AWS Glue ETL script

I am working on an AWS Glue ETL script using the dynamic frame glue abstraction and writing code in python. I created a JDBC connection resource named sap-lpr-connection in the glue data catalog and would like to use it to retrieve the connection…
LazyEval
  • 769
  • 1
  • 8
  • 22
1
vote
0 answers

Getting "javax.net.ssl.sslhandshakeexception pkix path building failed sun.security.provider.certpath" error in glue while accessing hive using jdbc

i am trying to access hive which is running on on-prmisis from a glue job As hive use jdbc connection which requires authentication, i attached certificate(certificate.pem) in the job(using connection) and trying to load the table with below…
cloud_hari
  • 147
  • 1
  • 8
1
vote
1 answer

Aws Glue Error ModuleNotFoundError: No module named 'pyodbc'

I want to connect to Microsoft SQL Server using python script which I will execute on AWS Glue. I added JOb parameter at Security configuration->script libraries->job parameters (optional) section as below : Key : --additional-python-modules Value:…
Sam
  • 392
  • 1
  • 6
  • 18
1
vote
2 answers

AWS Glue python shell - Using multiple libraries

I was using AWS glue python shell. The program uses multiple python libraries which not natively available for AWS. Glue can take .egg or .whl files for external library reference. All we need to do is put these .egg or .whl file in some S3 location…
Ludwig
  • 782
  • 1
  • 8
  • 24
1
vote
1 answer

AWS Glue assigned all tasks to the same worker

I have an AWS Glue job whose work is very simple: break large CSV gzip files into 1GB ones. In my test, I uploaded 4 files into the bucket, each is around 5GB. Yet, the job always assigns all files to a single worker instead of distributing across…
user1888955
  • 626
  • 1
  • 9
  • 27
1
vote
1 answer

glue PostgreSQL connection: “Unable to find suitable security group”

I am trying to add a new RDS connection for a PostgreSQL database. I am trying to follow this post to make this set up, but when trying to add the security group in the source field from inbound rules I get this message: You may not specify a…
brenda
  • 656
  • 8
  • 24
1
vote
1 answer

How do you specify Project ID in the AWS Glue to BigQuery connector?

I'm trying to use the AWS Glue connector to BigQuery following the tutorial in https://aws.amazon.com/blogs/big-data/migrating-data-from-google-bigquery-to-amazon-s3-using-aws-glue-custom-connectors/ but after following all steps I get a: :…
tonicebrian
  • 4,715
  • 5
  • 41
  • 65
1
2 3