How to do I run encrypted distcp from hdfs to s3?

Question

I like to copy data from our hadoop cluster (on premise) to s3. I can do it unencrypted. I can also run s3cmd put with client side encryption. How do I do distcp with client side encryption ?

Answer 1

Recent Hadoop versions should do the transfer using client side encryption just fine.

To be safe and not fall victim to specifics of your setup, I would simply ensure secure transfer by putting a bucket policy like the below one in place, forbidding non-secure transfer.

{
    "Statement":[
        {
            "Action": "s3:*",
            "Effect":"Deny",
            "Principal": "*",
            "Resource":"arn:aws:s3:::bucketname/*",
            "Condition":{
                "Bool":
                { "aws:SecureTransport": false }
            }
        }
    ]
}