I am working on Python flask REST Api to upload files to S3/ECS. My code is working perfectly fine and uploading files to S3/ECS. Here is the code...
def UploadFileToEcs(self,inputData,file):
BucketName = inputData.ecsbucketname
newpath = os.path.join(tempfile.gettempdir() , inputData.filename)
file.save(newpath)
s3_client = boto3.client('s3',endpoint_url=inputData.endpointUrl,
aws_access_key_id=inputData.access_key_id,
aws_secret_access_key=inputData.secret_key)
try:
newkey = inputData.folderpath + '/' + inputData.filename
response = s3_client.upload_file(newpath, BucketName,newkey)
except Exception as e:
return "error while uploading files to ECS" + str(e)
return True
The problem comes when the file size is too large like 8-10GB.
I am using PCF cloud and my container cannot go beyond 2GB RAM size. So my service fails in this scenario and give me error like...
Disk Quota exceed or No space available.
This is how I am receiving file on my rest api and calling ecsConnection to upload the file...
def UploadFileToEcs(self,request):
try:
inputData = self.getInputDataFromRequest(request)
if(inputData == None):
print('unable to retrieve request data. Terminating the request.')
raise BadRequest("unable to retrieve request data. Terminating the request.")
print('provided input is:= ' + inputData.getString())
file = request.files['file']
connector = EcsConnector()
isSuccessful = connector.UploadFileToEcs(inputData,file)
return str(isSuccessful)
except Exception as e:
print(str(e))
return str(e)
The question is: how can I call the service to accept file as stream and call S3 api to upload the stream. I believe with this approach I can reduce the memory footprint and upload even bigger size of file.
