0

Hello I am new here on AWS i was trying to upload a csv file on my bucket s3 but when the file is larger than 10mb it is returing "{"message":"Request Entity Too Large"}" I am using postman to do this. Below is the current code I created but in the future I will add some validation to change the name of the file that being uploaded into my format. Is there any way to do this with this kind of code or if you have any suggestion that can help me with the issue I have encountered?

   const AWS = require('aws-sdk');
const s3 = new AWS.S3();
const bucket = process.env.UploadBucket;
const prefix = "csv-files/";
const filename = "file.csv";


exports.handler = (event, context, callback) => {
    let data = event.body;
    let buff = new Buffer(data, 'base64');
    let text = buff.toString('ascii');
    
    console.log(text);
    
    let textFileSplit = text.split('?');
    //get filename split
    let getfilename = textFileSplit[0].split('"');
    
    console.log(textFileSplit[0]);
    console.log(textFileSplit[1]);

 //   //remove lower number on csv
    let csvFileSplit = textFileSplit[1].split('--')
    
    
    const params = {
        Bucket: bucket,
        Key: prefix + getfilename[3],
        Body: csvFileSplit[0]
    };

    s3.upload(params, function (err, data) {
        if (err) {
            console.log('error uploading');
            callback(err);
        }
        console.log("Uploaded")
        callback(null, "Success")
    });
}
Mina Chan
  • 41
  • 8

1 Answers1

2

For scenarios like this one, we normally use a different approach.

Instead of sending the file to lambda through API Gateway, you send the file directly to S3. This will make your solution more robust and cost you less because you don't need to transfer the data to API Gateway and you don't need to process the entire file inside the lambda.

The question is: How do you do this in a secure way, without opening your S3 Bucket to everyone on the internet and uploading anything to it? You use s3 signed urls. Signed Urls are a feature of S3 that allows you to bake in the url the correct permissions to upload an object to a secured bucket.

In summary the process will be:

  1. Frontend sends a request to API Gateway;
  2. API Gateway forward the request to a Lambda Function;
  3. The Lambda Function generate a signed Url with the permissions to upload the object to a specific s3 bucket;
  4. API Gateway sends back the response from Lambda Function to the Frontend. Frontend upload the file to the signed Url.

To generate the signed url you will need to use the normal aws-sdk in your lambda function. There you will call the method getSignedUrl (signature depends on your language). You can find more information about signed urls here.

Gustavo Tavares
  • 2,579
  • 15
  • 29
  • Thank you for you for this. I am using node.js on my lambda. Is there any sample where I can get idea on implementation – Mina Chan Aug 11 '20 at 10:20
  • https://stackoverflow.com/questions/38831829/nodejs-aws-sdk-s3-generate-presigned-url – Gustavo Tavares Aug 11 '20 at 10:53
  • one detail: if you are running this inside lambda you don't need to call AWS.config.update – Gustavo Tavares Aug 11 '20 at 10:53
  • Hello again i have Created a signedURL Lambda but when i am testing it on postman still file {"message":"Request Entity Too Large"} – Mina Chan Aug 12 '20 at 03:32
  • in that suggestion you don't send the file to the api gateway. You send the file to the signed url. Can you share your lambda code in github gist or in another answer here? – Gustavo Tavares Aug 12 '20 at 06:58