I try to use a Lambda function via an API Gateway to get a file from a folder within an S3 Bucket. I want to do this to import the latest version of a csv file in PowerBI/Tableau for data analysis. I'm able to do this if I insert the filename. however, this obviously doesn't resolve in the latest file. I want the code to always take the latest file in that folder. I'm planning on doing this via the Last Modified attribute, not via the filename itself.
my Bucket looks like this BUCKET -Input -input file.csv -Output -Output18042019.csv -Output19042019.csv
The code I have, however, doesn't allow for searching within folders (as far as I'm aware) and neither does it seem to take any latest file. I have tried putting a file in the root folder of the bucket to see if it works, but it doesn't. How can I solve the problem?
import json
import boto3
from datetime import datetime
def lambda_handler(event, context):
# TODO implement
get_last_modified = lambda obj: int(obj['LastModified'].strftime('%s'))
client = boto3.client('s3')
objs= client.list_objects_v2(Bucket='BUCKET')['Contents']
last_added = [obj['Key'] for obj in sorted(objs, key=get_last_modified)][0]
bucket='BUCKET'
link = client.generate_presigned_url('get_object', {'Bucket': bucket, 'Key': last_added}, 7200, 'GET')
return {
"statusCode": 303,
"headers": {'Location': link}
}
The Error I'm getting is the following:
Response:
{
"stackTrace": [
[
"/var/task/lambda_function.py",
11,
"lambda_handler",
"objs= client.list_objects_v2(Bucket='BUCKET')['Contents']"
],
[
"/var/runtime/botocore/client.py",
314,
"_api_call",
"return self._make_api_call(operation_name, kwargs)"
],
[
"/var/runtime/botocore/client.py",
612,
"_make_api_call",
"raise error_class(parsed_response, operation_name)"
]
],
"errorType": "ClientError",
"errorMessage": "An error occurred (AllAccessDisabled) when calling the ListObjectsV2 operation: All access to this object has been disabled"
}
Request ID:
"afd650be-4841-43cf-9cf4-731390bea1ce"
Function Logs:
START RequestId: afd650be-4841-43cf-9cf4-731390bea1ce Version: $LATEST
An error occurred (AllAccessDisabled) when calling the ListObjectsV2 operation: All access to this object has been disabled: ClientError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 11, in lambda_handler
objs= client.list_objects_v2(Bucket='BUCKET')['Contents']
File "/var/runtime/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
ClientError: An error occurred (AllAccessDisabled) when calling the ListObjectsV2 operation: All access to this object has been disabled
END RequestId: afd650be-4841-43cf-9cf4-731390bea1ce
REPORT RequestId: afd650be-4841-43cf-9cf4-731390bea1ce Duration: 2125.21 ms Billed Duration: 2200 ms Memory Size: 128 MB Max Memory Used: 58 MB
