Is there a way to see files stored in localstack's mocked S3 environment

Question

I've setup a localstack install based off the article How to fake AWS locally with LocalStack. I've tested copying a file up to the mocked S3 service and it works great.

I started looking for the test file I uploaded. I see there's an encoded version of the file I uploaded inside .localstack/data/s3_api_calls.json, but I can't find it anywhere else.

Given: DATA_DIR=/tmp/localstack/data I was expecting to find it there, but it's not.

It's not critical that I have access to it directly on the file system, but it would be nice.

My question is: Is there anywhere/way to see files that are uploaded to the localstack's mock S3 service?

Are you looking at /tmp/localstack/data on your local machine, or are you looking at that directory within the running docker container? You might need to check the docker-compose file to see where the container's /tmp/localstack/data path is located on your local machine -- it looks like the default is $TMPDIR/data. — Jason Rosendale, Jul 11 '19 at 20:20
The docker-compose.yml is setup with `DATA_DIR=/tmp/localstack/data`, and a volume of `'./.localstack:/tmp/localstack'`. I've checked both (first one on local, second inside docker) and the only thing there is the .json file. I also checked under $TMPDIR and there's a `localstack` directory, but it's empty. — Alan W. Smith, Jul 12 '19 at 13:13

Debu Shinobi · Answer 1 · 2023-03-02T08:04:59.157

32

After the latest update, now we have only one port which is 4566.

Yes, you can see your file. Open http://localhost:4566/your-bucket-name/you-file-name in chrome.

You should be able to see the content of your file now.

edited Mar 02 '23 at 08:04

answered Jan 12 '21 at 12:35

Debu Shinobi

2,057
18
21

1

Not sure why, but in my case, it returns 403. Just to access the bucket is ok http://localhost:4566/your-funny-backet-name But not able to see the content of the file. – Max Oct 12 '21 at 04:24
1

@Max permissions can be set at the bucket or object level. You should check the permissions on the object itself to see what it is. – Jimmy Vo May 08 '22 at 17:30

Alan W. Smith · Answer 2 · 2020-01-06T20:41:03.633

11

I went back and re-read the original article which states:

"Once we start uploading, we won't see new files appear in this directory. Instead, our uploads will be recorded in this file (s3_api_calls.json) as raw data."

So, it appears there isn't a direct way to see the files.

However, the Commandeer app provides a view into localstack that includes a directory listing of the mocked S3 buckets. ~~There isn't currently a way to see the contents of the files, but the directory structure is enough for what I'm doing.~~ UPDATE: According to @WallMobile it's now possible to see the contents of files too.

edited Jan 06 '20 at 20:41

answered Jul 12 '19 at 13:18

Alan W. Smith

24,647
4
70
96

5

You can now view the contents of the files in the latest version of Commandeer. – WallMobile Sep 05 '19 at 21:38
added a new image below, as this was a very preliminary version of Commandeer. Alan - I'll remove that item below, if you update your screenshot to that one. Thanks, Bob – WallMobile Jan 03 '20 at 19:15
@WallMobile right on. Just updated the screenshot and updated the text to reflect your comment about it now being possible to see files. Thanks! – Alan W. Smith Jan 06 '20 at 20:42
Awesome. Are you using the tool much? We have pushed out a lot of new features recently, so any feedback we always love to hear it. – WallMobile Jan 06 '20 at 20:52
@WallMobile - I haven't been doing much AWS dev work recently. I'll be sure to let you know when I get back into it. – Alan W. Smith Jan 06 '20 at 21:46
Do we need the `s3_api_calls.json` file? – larueroad Aug 10 '20 at 21:19
@AlanW.Smith This is a great tool. Thanks a lot! – Andrew Quartey Dec 29 '20 at 16:27

score 9 · Answer 3 · answered Aug 30 '19 at 09:57

9

You could use the following command

aws --endpoint-url=http://localhost:4572 s3 ls s3:<your-bucket-name>

answered Aug 30 '19 at 09:57

Yuvi

528
8
18

7

It's just `aws --endpoint-url=http://localhost:4572 s3 ls ` – bovus Jan 14 '20 at 21:43
4

As of the most recent update, you should be using the unified port `4566`. So it would be `aws --endpoint-url=http://localhost:4566 s3 ls ` – jdenoc Jun 16 '21 at 01:10

score 0 · Answer 4 · edited May 19 '22 at 18:37

Image is saved as base64 encoded string in the file recorded_api_calls.json

I have passed DATA_DIR=/tmp/localstack/data

and the file is saved at /tmp/localstack/data/recorded_api_calls.json

Open the file and copy the data (d) from any API call that looks like this

"a": "s3", "m": "PUT", "p": "/bucket-name/foo.png"

extract this data using bas64 decoding

You can use this script to extract data from localstack s3

https://github.com/nkalra0123/extract-data-from-localstack-s3/blob/main/script.sh

score 0 · Answer 5 · answered Sep 17 '21 at 09:56

0

In order to list the exact folder in s3 bucket you could use this command:

aws --endpoint-url=http://localhost:4566 s3 ls s3://<bucket-name>/<folder-in-bucket>/

answered Sep 17 '21 at 09:56

Aleksandar

636
1
7
24

Alexis Wilke · Answer 6 · 2022-05-27T18:32:33.223

From my understanding, localstack saves the data in memory by default. This is what happens unless you specify a data directory. Obvious, if in memory, you won't see any files anywhere.

To create a data directory you can run a command such as:

mkdir ~/.localstack

Then you have to instruct localstack to save the data at that location. You can do so by adding the DATA_DIR=... path and a volume in your docker-compose.yml file like so:

localstack:
  image: localstack/localstack:latest
  ports:
    - 4566:4566
    - 8055:8080
  environment:
    - SERVICES=s3
    - DATA_DIR=/tmp/localstack/data
    - DOCKER_HOST=unix:///var/run/docker.sock
  volumes:
    - /home/alexis/.localstack:/tmp/localstack

Then rebuild and start the docker.

Once the localstack process started, you'll see a JSON database under ~/.localstack/data/....

WARNING: if you are dealing with very large files (Gb), then it is going to be DEAD SLOW. The issue is that all the data is going to be saved inside that one JSON file in base64. In other words, it's going to generate a file much bigger than what you sent and re-reading it is also going to be enormous. It may be possible to fix this issue by setting the legacy storage mechanism to false:

LEGACY_PERSISTENCE=false

I have not tested that flag (yet).

Is there a way to see files stored in localstack's mocked S3 environment

6 Answers6