1

I am trying to create keys in apache OZone using AWS S3 API for .NET.

The key I am trying to create must be inside a bucket called "test" that I created using AWS S3 CLI.

My code:

    static async Task WriteFile()
    {
        AmazonS3Config config = new AmazonS3Config();
        config.ServiceURL = "http://myApacheOzoneEndpoint:8744"; // This port is mapped from a docker container to (not the original endpoint port for Ozone)

        AWSCredentials credentials = new BasicAWSCredentials("testuser/scm@EXAMPLE.COM", "c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999"); // Credentials must be set but can be random since Ozone doesn't use authentication
        AmazonS3Client client = new AmazonS3Client(credentials, config);

        using (FileStream fs = File.OpenRead(@"C:\Users\me\path.to.file\image.jpg"))
        {
            string responseBody = "";
            try
            {
                PutObjectRequest request = new PutObjectRequest
                {

                    BucketName = "test",
                    Key = "deleteme.jpg",
                    InputStream = fs
                };

                PutObjectResponse response = await client.PutObjectAsync(request);
                Console.WriteLine($"Result: {response.HttpStatusCode.ToString()}");
            }
            catch (AmazonS3Exception e)
            {
                Console.WriteLine("Error encountered ***. Message:'{0}' when writing an object", e.Message);
            }
            catch (Exception e)
            {
                Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message);
            }
        }
    }

This code is returning a 200 code error (OK), and if I call GetObjectAsync from the API I get an HTML with metadata as result (But can't read the file content by the moment)

Then I go to Apache Ozone and execute these commands in within the ozone shell:

bash-4.2$ ozone s3  path test
Volume name for S3Bucket is : s3c89e813c80ffcea9543004d57b2a1239
Ozone FileSystem Uri is : o3fs://test.s3c89e813c80ffcea9543004d57b2a1239
bash-4.2$ ozone sh bucket list /s3c89e813c80ffcea9543004d57b2a1239
[ {
  "volumeName" : "s3c89e813c80ffcea9543004d57b2a1239",
  "bucketName" : "test",
  "createdOn" : "Wed, 01 Apr 2020 08:06:40 GMT",
  "acls" : null,
  "versioning" : "DISABLED",
  "storageType" : "DISK",
  "encryptionKeyName" : "N/A"
}, {
  "volumeName" : "s3c89e813c80ffcea9543004d57b2a1239",
  "bucketName" : "deleteme.jpg",
  "createdOn" : "Tue, 31 Mar 2020 10:57:26 GMT",
  "acls" : null,
  "versioning" : "DISABLED",
  "storageType" : "DISK",
  "encryptionKeyName" : "N/A"
} ]
bash-4.2$ ozone sh key list /s3c89e813c80ffcea9543004d57b2a1239/test => This command returns only keys that have been put from AWS S3 CLI

As you can see, the object is created as a new bucket inside the Ozone volume used for S3, instead of being created as a new key below testbucket. If I try to put keys from AWS S3 Cli it works as expected

What is happening?

FYI I made the same example from Java API and the same issue occurs!

Thanks

dhalfageme
  • 1,444
  • 4
  • 21
  • 42
  • is it possible to run the ozone CLI from outside the cluster? Like one would do with aws cli, with which you can do operations on buckets from local machine. If it is possible to run the Ozone CLI outside a cluster (without connecting to OM host for example) is there a site where I can see how to build the CLI? Thanks – Jorge Cespedes Apr 22 '20 at 14:40

1 Answers1

3

Ozone s3-gateway uses by default the path-style addressing while updated sdk libraries use the virtual-hosted addressing. The quickest solution would be to switch to path-style:

// AmazonS3Config config = new AmazonS3Config();
config.ForcePathStyle = true;

Alternatively, as mentioned on the docs, you could enable the virtual-hosted schema in ozone.

Please notice that the path-style is going to be deprecated in aws s3.

danidelvalle
  • 146
  • 1
  • 7