how to keep reading a large file from s3 quickly

Question

I'm a beginner to aws s3 and I am trying to read a pretty large file (250mb) from S3 using JavaScript when I run localhost:8000.

I followed examples and wrote code that should retrieve data. But so far, it doesn't even go inside s3.getObject() function (no error message, no data returned). I think there's something failing right when it tries s3.getObject(). I'm not sure if it's because of authorization problem or if the file is too big. Or is there're any other issues I'm not aware of?

More specifically, I've been using the following code. s3.getObject() is called inside a middleware. I do get print messages of params: but do not get a print message of --- err: or --- data:, which indicates that there's something failing right when it tries s3.getObject.

const aws = require('aws-sdk');
const s3 = new aws.S3({ accessKeyId: 'directly wrote ID for now', secretAccessKey: 'directly wrote secretKey' });

module.exports.getData = (req, res, next) => {
  let getParams = {
    Bucket: 'bucket-name',
    Key: 'file-name'
  };

  console.log('params: ', getParams);
  s3.getObject(getParams, function(err, data) {
    console.log('--- err: ', err);
    console.log('--- data: ', data);
    req.myData = data;
    next();
  });
};

I want to see either error message or data inside s3.getObject().

Update

I added a test file that's very small and now I was able to read the file. So the problem is due to the size of the file... Any recommendation to handle large file reading?

You might want to compare your code with: [How to get response from S3 getObject in Node.js?](https://stackoverflow.com/a/36944450/174777) — John Rotenstein, Apr 15 '19 at 02:38
@JohnRotenstein Thank you for the link! They look pretty much the same to me. — harumomo503, Apr 15 '19 at 02:47
Hang on... you are defining `getData`, but you aren't actually calling it! — John Rotenstein, Apr 15 '19 at 02:49
I do call `getData` in another file.. that's why I get print messages of `params: ` (you see `console.log()` right before `s3.getObject`). I also mentioned in the question :] — harumomo503, Apr 15 '19 at 02:51

score 0 · Answer 1 · answered Apr 15 '19 at 02:33

0

Just a quick high level Troubleshooting tip here. Check in S3 that you have explicit permissions on the bucket. Even if you have a global super admin account, you still need to explicitly allow this user read/write access in the bucket. Another tip, not sure it’s applicable in your case, but I used some lower level tools that required a cli component to be installed on the machine and the access credentials + the region configuration [default] us-east-1 or whatever it is for you to be set correctly.

answered Apr 15 '19 at 02:33

George Mogilevsky

599
5
8

Thank you for the comment. But how do I check that I have explicit permissions on the bucket exactly? – harumomo503 Apr 15 '19 at 02:49
You need to open S3 bucket that you are targeting and look in the permissions. Add permissions for the user for which you created credentials. Go in the AWS Ui and in S3 find your bucket. Open it and look for permissions – George Mogilevsky Apr 15 '19 at 02:51

how to keep reading a large file from s3 quickly

Update

1 Answers1