5

I am trying to use Julia on an AWS computing cluster, processing data from Amazon S3 buckets.

How do I use in Julia AWSS3.jl to open a connection to an S3 bucket?

In particular I would like to know what is the best approach to configure the connectivity and subsequently use that configuration from Julia.

Przemyslaw Szufel
  • 40,002
  • 3
  • 32
  • 62
user2405703
  • 87
  • 1
  • 6
  • ...sketchy. i get the feeling aws cli was not installed so you were installing packages to hack – Kermit Aug 29 '21 at 20:06

1 Answers1

3

  1. Create an AWS policy with S3 access rights (in particular PutObject, GetObject etc.)

  2. Create an EC2 oriented IAM role and add the policy from the first step to that role

  3. Assign the IAM role to the EC2 instance (or perhaps configure it to use in an instance profile when creating EC2 instances for your cluster)

  4. Now you are ready to do your work in Julia. Below a simple example that serializes and deserializes any Julia object to an S3 bucket.

using AWS, AWSS3, Serialization
struct SampleData
  a::Int
  b::String
end

d=SampleData(1,"sss")
aws = global_aws_config(; region="us-east-1")
b = IOBuffer()
serialize(b, d)

s3_put(aws, "your-s3-bucket-name","myfile.bin", b.data)

ddat = s3_get(aws, "your-s3-bucket-name","myfile.bin")
d2 = deserialize(IOBuffer(ddat))

@assert d == d2

Should you have any problems with points 1-3, here is the tutorial: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-access-s3-bucket/

Przemyslaw Szufel
  • 40,002
  • 3
  • 32
  • 62
  • Many thanks, As a matter of fact, for others, I was using functions which for some reasons are not working such as s3_list_buckets. Hope this helps others – user2405703 Jun 09 '21 at 12:56