Upload data to s3 with curl

Question

I am recording video data with a HELO device. With a curl command, I can upload data to my local computer:

curl -v0 --output example-filename-downloaded.mov http://192.168.0.2/media0/example-filename-source.mov

Where 192.168.0.2 is replaced by the IP address that the device is connected with. Now, I want to download this data not to my own pc, but to a cloud environment (AWS S3). Normally when I upload data to s3, I use the aws s3 cp filename s3://bucketname/directory command. However, I want to set something up so that the file does not have to stored on the pc, but is uploaded to s3 immediately. So as if the curl command would have the s3 destination in it.

Any thoughts on how to do this?

Answer 1

There are two ways to upload data to Amazon S3:

• Use an AWS SDK that communicates directly with the S3 API, or
• Uploading a File to Amazon S3 Using HTTP POST

You can apparently Use cURL to upload POST data with files - Stack Overflow. I haven't tried it with S3, but it might work!

You would need to include authentication information in the headers, since you don't want to open your bucket to anyone being able to upload to the S3 bucket (otherwise bots will upload random movies and music, costing you money). This will make the curl command more complex since you need to provide the required information to make S3 happy.

A better method

Frankly, I suggest you use the AWS Command-Line Interface (CLI) since it will be much simpler. This, of course, requires the ability to install the AWS CLI on the system. If this is not the case for your HELO device, then you'll need to use the above method (good luck!).

If your concern is that "the file does not have to stored on the pc", then please note that your curl method will still involve downloading the file to your computer and then uploading to S3. Yes, the file is not saved to disk, but it still needs to pass through your computer.

If you can install the AWS CLI, then the equivalent method using the AWS CLI would be:

curl http://example.com/some.file.on.the.internet.txt | aws s3 cp - s3://my-bucket/filename.txt

This will retrieve a file from a given URL, then pass it via standard input (using - as the source) to the Amazon S3 bucket. The file will 'pass through' your computer, but will not be saved to disk. This is equivalent to the curl method you were wanting to do, but makes the copy to S3 much easier.

This can be useful if you want to retrieve a file from somewhere on the Internet and then upload it to S3. If your HELO device makes it possible to 'pull' a file, this could be done from your own PC.