C# with Google Storage - Signed URL with customer encryption key

Question

I uploaded a file to Google storage bucket. Now, I generated a Signed Url of it (based on this solution)

private static Uri SignUrl(Stream jsonCertificateStream, string bucketName, string objectName, TimeSpan expireAfter, string[] scopes, HttpMethod verb)
{
    string url;
    var urlSigner = UrlSigner.FromServiceAccountData(jsonCertificateStream);
    url = urlSigner.Sign(
                bucketName,
                objectName,
                expireAfter,
                verb);

    return new Uri(url);
}

I got back an URI which is working good with my browser. Up to here- everything working nice.

Now, I'm using Customer Encryption Key to encrypt my uploaded file. So, I removed the file and uploaded it again - now it's encrypted.

The problem is that now the SignUrl() method is not working any more.

When testing the outcome of the SignUrl() method with browser - I'm getting:

<Error>
    <Code>ResourceIsEncryptedWithCustomerEncryptionKey</Code>
    <Message>
        The resource is encrypted with a customer encryption key.
    </Message>
    <Details>
        The requested object is encrypted by a customer-supplied encryption key.
    </Details>
</Error>

I guess it's because my file is encrypted other key.

How to do it right?

Answer 1

Base on Customer-Supplied Encryption Keys documentation , you need to add the following headers when using Customer-Supplied Encryption Keys

+------------------------------+--------+------------------------------------------------------------------------------+
|         Header name          | Value  |                                 Description                                  |
+------------------------------+--------+------------------------------------------------------------------------------+
| x-goog-encryption-algorithm  | string | The encryption algorithm to use. You must use the value AES256.              |
| x-goog-encryption-key        | string | An RFC 4648 Base64-encoded string of your AES-256 encryption key.            |
| x-goog-encryption-key-sha256 | string | An RFC 4648 Base64-encoded string of the SHA256 hash of your encryption key. |
+------------------------------+--------+------------------------------------------------------------------------------+

also when using urlSigner you have to pass x-goog-encryption-algorithm

here a code example

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Google.Cloud.Storage.V1;
using System.IO;
using System.Net.Http;
using Google.Apis.Auth.OAuth2;

namespace stackoverflow54387198
{
    class Program
    {
        static void Main(string[] args)
        {

            // step 1 create customer encryption key
            var key = EncryptionKey.Generate().Base64Key;

            var encryptionKey = EncryptionKey.Create(Convert.FromBase64String(key));


            // step 2 get you service Acount cert for auth

            string serviceAcountCert = "stackoverflow54387198-xxxxxxxx.json";

            // step 3 get you service Acount cert for auth

            string bucketName = "stackoverflow_54387198_bucket";

            string localPath = "FileToUpload.txt";

            string objectName = null;


            // step 4 create a local text file to upload

            File.WriteAllText(localPath, "test");


            // step 5 Create Google Storage Client

            var storage = StorageClient.Create(
                GoogleCredential.FromJson(File.ReadAllText(serviceAcountCert)));


            // step 6 upload the file with the customer encryption key from step 1

            using (var f = File.OpenRead(localPath))

            {

                objectName = objectName ?? Path.GetFileName(localPath);

                storage.UploadObject(bucketName, objectName, null, f,

                    new UploadObjectOptions()

                    {

                        EncryptionKey = encryptionKey

                    });

                Console.WriteLine($"Uploaded {objectName}.");

            }


            // step 7 create a url


            // step 7.1 create add x-goog-encryption-algorithm hear 
            //to tell google you are using  customer encryption key

            var requestHeaders = new Dictionary<string, IEnumerable<string>>
            {
                {
                    "x-goog-encryption-algorithm", new [] { "AES256" }
                }
            };


            // step 7.2  set other parameters

            var expireAfter = TimeSpan.FromHours(30.0);

            var verb = HttpMethod.Get;


            // step 7.3  create a Url Signer

            var urlSigner = UrlSigner.FromServiceAccountPath(serviceAcountCert);


            // step 7.4  create a secure url
            var url = urlSigner.Sign(
                        bucketName,
                        objectName,
                        expireAfter,
                        verb,
                        requestHeaders);



            // step 8  use the Your Url


            // step 8.1 create HttpClient

            var client = new HttpClient();


            // step 8.1  add x-goog-encryption-algorithm header the same from step 7

            client.DefaultRequestHeaders.Add("x-goog-encryption-algorithm", "AES256");


            // step 8.2  add x-goog-encryption-key header with customer encryption key (Base64Hash)

            client.DefaultRequestHeaders.Add("x-goog-encryption-key", encryptionKey.Base64Key);


            // step 8.3  add x-goog-encryption-key header with customer encryption key (Base64Hash)

            client.DefaultRequestHeaders.Add("x-goog-encryption-key-sha256", encryptionKey.Base64Hash);

            // step 8.4  Download the file 
            Task.Run(async () =>
            {
                var response = await client.GetAsync(url);
                var contents = await response.Content.ReadAsStringAsync();
                // contents == "test"
                Console.WriteLine($"contents=>{contents}");

            }).GetAwaiter().GetResult();

            Console.ReadLine();

        }


    }
}

also you can use postman to include the headers

ref. 1

• Generating Signed URLs with Your Own Program  |  Cloud Storage  |  Google Cloud

• Using Customer-Supplied Encryption Keys  |  Cloud Storage  |  Google Cloud

• Creating Storage Buckets  |  Cloud Storage  |  Google Cloud

• GCS Signed URL with Customer Supplied Encryption Key

• dotnet-docs-samples/Storage.cs at master · GoogleCloudPlatform/dotnet-docs-samples

• c# - Adding headers when using httpClient.GetAsync - Stack Overflow

• c# - Can't specify the 'async' modifier on the 'Main' method of a console app - Stack Overflow

• How to put tables in Stack Overflow? - Meta Stack Overflow

Extra

Hi Mohamed, thanks for the comprehensive answer. According to your answer, I understand that I cant just make a public signed URL that will be used to download the file. I always must add additional headers (x-goog-encryption-algorithm, x-goog-encryption-key, ...) in order to get the file? It's isn't possible to include this data in the URL field somehow?

To answer this i will use the flowing Questions :

• skip to question 3 for solution

Question 1

It's isn't possible to include this data in the URL field somehow ?

Answer 1

• I don't know

Why Answer 1 ?

• base on my Logic

• the google Storage API is a software written by a google programmer

• from the documentation they expect the encryption key to be in the header

• If you call the API without the key in header it will return an error message

• so maybe there is undocumented way , but I don't know

Question 2

What is the difference between HTTP parameters and HTTP headers usage

Answer 2

• see What is the difference between HTTP parameters and HTTP headers?

Question 3 :

How to hack around this ?

Answer 3

• if your are using this URL in a web page see : javascript - How to set a header for a HTTP GET request, and trigger file download? - Stack Overflow

• if your are using this URL in mobile app : you can download the file using the swift or java or the language of your app

• if your are using this URL in an email : you can create your own REST API using asp.net MVC or API which accept the encryption key as parameter and then you act like a proxy between the google could and the client browser

Question 4 :

What is google storage Data Encryption Options

Answer 4

Cloud Storage always encrypts your data on the server side, before it is written to disk, at no additional charge. Besides this standard behavior, there are additional ways to encrypt your data when using Cloud Storage. Below is a summary of the encryption options available to you:

• Server-side encryption: encryption that occurs after Cloud Storage receives your data, but before the data is written to disk and stored.

  • Customer-supplied encryption keys: You can create and manage your own encryption keys for server-side encryption, which act as an additional encryption layer on top of the standard Cloud Storage encryption.
  • Customer-managed encryption keys: You can generate and manage your encryption keys using Cloud Key Management Service, which act as an additional encryption layer on top of the standard Cloud Storage encryption.

• Client-side encryption: encryption that occurs before data is sent to Cloud Storage. Such data arrives at Cloud Storage already encrypted but also undergoes server-side encryption.

• ref. : Data Encryption Options  |  Cloud Storage  |  Google Cloud

Question 5 :

Is better to use Customer-Supplied Encryption Keys ?

Answer 5 :

• it depends on you software scenario , but

  Cloud Storage always encrypts your data on the server side, before it is written to disk, at no additional charge

• ref: Data Encryption Options  |  Cloud Storage  |  Google Cloud

• Maybe you don't need to encrypt your data using Customer-supplied encryption keys , because google encrypt data at rest by default, also by the end of the day if you afraid of google spying on you (just my assumptionsas as a perfectionist myself ) ,it's not logical , because your are sending the encryption key to there servers when you request the file

ref. 2 :

• Mark It Down: convert rich text to Markdown