3

I am using Boto3 to read the results of my Athena query in a python script.

I have the following code that works fine in AWS Lambda.

def get_athena_results(s3_bucket, s3_output_path, execution_id):
    s3client = boto3.client('s3')
    key = s3_output_path + '/' + execution_id + '.csv'
    obj = s3client.get_object(Bucket=s3_bucket, Key=key)
    results_iterator = obj['Body'].iter_lines()
    results = [r for r in results_iterator]
    return results

When I run the same function in AWS Glue Python Shell (Not a Spark job), I get the error:

Unexpected error: <class 'AttributeError'>
'StreamingBody' object has no attribute 'iter_lines'

This doesn't make sense to me as the botocore.response.StreamingBody class has an iter_lines method, and it works fine in AWS Lambda.

https://botocore.amazonaws.com/v1/documentation/api/latest/reference/response.html

Any idea why this is happening in AWS Glue Python Shell?

Thanks

Diego Serrano
  • 846
  • 2
  • 15
  • 34
  • Check the version of boto3 package in Glue – Lamanus Sep 05 '19 at 01:54
  • This issue seems to be happening because of version mismatch. Can you print versions in your Glue job,Lambda function and compare ? print("boto3 version:"+boto3.__version__) print("botocore version:"+botocore.__version__) – Prabhakar Reddy Sep 05 '19 at 12:58
  • Hi, I used your code @bdcloud and I am seeing `boto3 version:1.9.130`. It seems like Glue's Boto is not the last available. https://stackoverflow.com/questions/52972342/update-boto3-for-aws-glue I tried updating it using subprocesses but run with user permission issues. Any ideas how to fix it? – Diego Serrano Sep 05 '19 at 16:35
  • @DiegoSerrano Can you try installing the version which is matching lambda in your Glue python shell. Please refer to https://stackoverflow.com/a/54852126/4326922 and mention package as boto3==1.9.130(version from lambda) and let me know how it goes – Prabhakar Reddy Sep 05 '19 at 16:44
  • Have you tried the suggestion provided? Did it helped? – Prabhakar Reddy Sep 10 '19 at 15:11

1 Answers1

0

The error happened as, at the time posting the question, Glue's Boto3 is on a previous version VS the one available in lambda, where iter_lines() is unavailable.

Diego Serrano
  • 846
  • 2
  • 15
  • 34