-2

I have a json file on AWS s3. I want to download them into a struct, not into a file. is this possible? or a way to delete the file after i put the data into a struct? i dont want extra files generated in my application.

I see the go example AWS s3 provides here

and reading the go aws s3 docs on the download func it writes to io.WriterAt is "io.writerAt" only can be a file type?

my problem is when i read S3 json files, and load my data into a struct, but it also ends up creating files in the application due to what i think io.writerAt must be a file type.

heres my code that reads in the AWS s3 json file, creates a file(which i dont want), and loads it into struct to pass along the rest of the code.

sess, _ := session.NewSession(&aws.Config{
    Region: aws.String(c.AwsLocation),
})
downloader := s3manager.NewDownloader(sess)

doc := "myfile.json" 

file, err := os.Create(doc) //PROBLEM creates file.
//todo err
defer file.Close()

numBytes, err := downloader.Download(file,
    &s3.GetObjectInput{
        Bucket: aws.String(c.AwsBucket),
        Key:    aws.String(doc),
    })
//todo err
fmt.Println("Downloaded", file.Name(), numBytes, "bytes")

//take data, and put in struct
byteValue, _ := ioutil.ReadAll(file)
var output myStruct
json.Unmarshal(byteValue, &output)
return output
not_fubar_yet
  • 194
  • 15
  • 1
    I started to figure this out and quickly found the duplicate answer above by seeing that the Download method wanted a io.WriterAt as a first parameter, then googling for it – Vorsprung Dec 11 '18 at 19:38
  • thanks! i really did try to google it but failed to use "buffer". my searches "golang aws download to struct, not file" or variants didnt show up this. i consider myself a novice and dont know the proper terms yet. – not_fubar_yet Dec 11 '18 at 20:42
  • The AWS documentation is very good and is a great place to take a look if you are stuck – Vorsprung Dec 11 '18 at 20:43
  • @vorsprung. yup aws docs generally are great. thats where i started. i even linked it in the 1st place, but I was too dumb to implement that buffer you showed me, which isnt in aws docs. – not_fubar_yet Dec 12 '18 at 00:55

1 Answers1

0

It looks like golang does have a pickle equivalent so to speak, https://golang.org/pkg/encoding/gob/.

But I'd use JSON to do this if at all possible. You can unmarshal the json into your struct pretty easily (https://gist.github.com/dmikalova/5693142 for example). More importantly, by choosing a more generic storage format, you'll be able to read the files from any language and your data will be distinct from your in-application model, which is a very friendly approach for cloud or micro- services.

erik258
  • 14,701
  • 2
  • 25
  • 31
  • thanks. i dont have a problem with json marshalling... my problem is the s3 download function seems to require that i create an in app file. Is there a way to skip that and download straight to my struct? – not_fubar_yet Dec 11 '18 at 18:29