Fetch non-public data from S3

Question

I have been working on a webapplication, where I scrape data by using Scrapy and launch the data on S3. Now I want to fetch the data to my React project. This is working well, if I set the data to be public.

axios
  .get(`https://s3-eu-west-1.amazonaws.com/bucket/data.json`)
  .then(res => {
    console.log("Data: ", res.data);
    this.setState({ events: res.data });
  })
  .catch(console.log("error"));

Question

I don't want the data I'm scraping to be public and should only be available for my webapplication. Is this even possible?

Answer 1

I'm assuming you're talking about a client-side web application that runs in the user's browser? As far as I know, you need at least some server-side component to control or allow access to private S3 resources. This could be a lambda function or an actual server, but AFAIK there is no safe way to do this from the client only.

There are two ways that I'm aware of to expose private S3 resources to a client-side app:

1. If there is a server under your control (EG a NodeJS server that delivers your app, or perhaps provides API services) you can connect to S3 securely from the server side and deliver whatever you need to the client side. This could also be done from a lambda function. Whatever you choose you still need a way to make sure the client/app requesting the content should have access to the content, EG the user should have a valid session.

2. You could allow access to a private S3 object by generating a pre-signed URL that gives the client app some fixed amount of time to download the content. This is probably an endpoint on your server (or lambda) that your client-side app calls only after making sure the user that requested it is authorized.

Here's a tutorial on Medium that explains both options: https://blog.fyle.in/sharing-files-using-s3-pre-signed-urls-e05d4603e067

Here's a StackOverflow answer with example code for Node: Nodejs AWS SDK S3 Generate Presigned URL