1

I am trying to stream the data in the retrieved s3 file into a local file on disk. However, when I try my current code, I do not get a file stored on my computer, or in any location near the code.

I first request the object from s3 using a getObjectOutcome. After success, I want to create an ofstream and redirect the objects stream buffer to my local object so that I can create a file on disc. However, the following code does not create a file on my computer. What am I doing wrong?

Here is the get object function:

bool GetObject(const Aws::String& objectKey,
    const Aws::String& fromBucket,
    const Aws::Client::ClientConfiguration& clientConfig) {
    Aws::S3::S3Client client(clientConfig);

    Aws::S3::Model::GetObjectRequest request;
    request.SetBucket(fromBucket);
    request.SetKey(objectKey);

    Aws::S3::Model::GetObjectOutcome outcome =
        client.GetObject(request);

    if (!outcome.IsSuccess()) {
        const Aws::S3::S3Error& err = outcome.GetError();
        std::cerr << "Error: GetObject: " <<
            err.GetExceptionName() << ": " << err.GetMessage() << std::endl;
    }
    else {
        std::cout << "Successfully retrieved '" << objectKey << "' from '"
            << fromBucket << "'." << std::endl;
        std::ofstream localfile;
        localfile.open(objectKey.c_str(), std::ios::out | std::ios::binary);
        auto retrieved = outcome.GetResult().GetBody().rdbuf();
        localfile << retrieved; 
        std::cout << "Done!";
    }

    return outcome.IsSuccess();
}

Here is an image of the memory for local file and retrieved: image of output

Would someone teach me what I am doing this wrong, or how to correctly download data from s3 to disc?

Thanks.

I tried downloading some data from s3 to disc. I am having trouble outputting this data via stream buffer to local file. I have been looking online and cannot find a similar problem.

Update:

I am now on my second day of trying to figure this out to no avail. For some reason, the code will not even output a file after it has been created to the directory I have set up for the .nc files to be written to.

I have tried the following updates:

bool GetObject(const Aws::String& objectKey,
    const Aws::String& fromBucket,
    const Aws::Client::ClientConfiguration& clientConfig) {
    Aws::S3::S3Client client(clientConfig);

    Aws::S3::Model::GetObjectRequest request;
    request.SetBucket(fromBucket);
    request.SetKey(objectKey);

    Aws::S3::Model::GetObjectOutcome outcome =
        client.GetObject(request);

    if (!outcome.IsSuccess()) {
        const Aws::S3::S3Error& err = outcome.GetError();
        std::cerr << "Error: GetObject: " <<
            err.GetExceptionName() << ": " << err.GetMessage() << std::endl;
    }
    else {
        std::cout << "Successfully retrieved '" << objectKey << "' from '"
            << fromBucket << "'." << std::endl;

        //create the filename, which will be the objectKey 
        std::string local_file_name = "./netcdf/" + objectKey;
        std::ofstream local_file(local_file_name, std::ios::binary);
        auto &retrieved = outcome.GetResult().GetBody();
        local_file << retrieved.rdbuf();
        std::cout << "Done!";

    }

    return outcome.IsSuccess();
}

Then, after opening the ./netcdf folder, there is no output.

Here is the file structure inside my project for reference with the code:

filestructure

I am still confused as to what I need to do here.

Thank you for all of the help you can offer!

Corey4005
  • 45
  • 7

1 Answers1

1

You are using a folder with "./" at the front. This means that the file will be relative to the current working directory (cwd) of the binary. That is likely not the src folder

Just to get past your problem, use a full absolute path to see if the rest of your code works.

Also, try adding

 // You need "#include <filesystem>" for the next line
 cout << std::filesystem::current_path();

To see where the files you made might be

Lou Franco
  • 87,846
  • 14
  • 132
  • 192