Uploading to Google Cloud using a signed URL

Question

I'm trying to generate download and upload link from the Google Cloud, to view and upload files, using the following code:

public class Test {

public static void main(String[] args) throws IOException {
Storage storage = StorageOptions.newBuilder().setCredentials(
    ServiceAccountCredentials.fromStream(new FileInputStream("C:/cred/Key.json")))
    .build()
    .getService();

String filePath = "file/path/";
File file = new File(filePath);
byte[] bytes = Utilities.fileToByteArray(file);
String mimeType = Utilities.getMimeType(bytes);
BlobId blobId = BlobId.of("bucket", file.getName());
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).setContentType(mimeType).build();
URL urlGet = storage
    .signUrl(BlobInfo.newBuilder("bucket", "object").build(), 1, TimeUnit.HOURS,
        SignUrlOption.httpMethod(HttpMethod.GET));
URL urlPut = storage
    .signUrl(blobInfo, 1, TimeUnit.DAYS, SignUrlOption.httpMethod(HttpMethod.PUT),
        SignUrlOption.withContentType());
System.out.println(urlGet);
System.out.println(urlPut);
  }

}

urlGet contains the download link and urlPut contains the upload link. When I run the program, I get the following output:

https://storage.googleapis.com/roshanbucket/jasperviewpdf?GoogleAccessId=myservice@deft-idiom-234709.iam.gserviceaccount.com&Expires=1552986620&Signature=OZl6M4uMkigu6JBMYoDumgs8P4EC%2BtcK44zHMPkG2xzq3hFfsrj8YYRRtajI8diz64vdCX54nct3wuEpXNRwcnzCmq4KdD53%2B8gbERNuttm8K6%2BlZDLGF3wng%2BCSMzghbGbLnYaZRiZbvjCG%2B3ObBUg9ZiY0qRlif9nyGFftsGUF9GGHvHP6HWP51DJOAurGytSjf9SA5HKPOw4e%2B%2BP1LltfI7m3WjWhxwnSYz4lVxcR4eksec7ILTi66jnwu1gxXtqp75XTxLp9vQa6RC4dCPGoTridFQcMqm89TVzf58c8krk7apQCR6TWp2tAWuFr2xJ1U5FwFfiBaoSX4A33rw%3D%3D

https://storage.googleapis.com/roshanbucket/pi.jpg?GoogleAccessId=myservice@deft-idiom-234709.iam.gserviceaccount.com&Expires=1553069420&Signature=YHsGTgXIBum9t5M7U%2F9fdibDvzBKttQGh0jxzbYgJkevQbpOh8gRQYOlHdjT86byobDE5TNEGF5VrGFAtI5rhRGxLw0xqcNT%2BYGfvHxAIfAJXy5ormXxWVnVEnwGMafyVLOtdIY4asa0niFu%2B36eaIqtD5UzsjUY%2F18OW%2FwvjfQmhlmsvJ7qSkfD1Oif5Rv6c%2F67z1zT7gz7rB4gTCG6mLALuRrOIwCPO%2BkyzOxP9PhEJkoB7j446v%2BhE%2F0pt0wM2nJ29%2BK3HRUShhccJzzZ%2BZRxgOXeUL44CsnYlssaTThU%2FztyUbsXWXbs2hroTcFxVVtOp7aGeCUs1qjdJkXaEg%3D%3D

When I click on the first link (i.e download), it renders the file from the bucket without any problem, but when I use the second link to upload a file from my computer to the Google Cloud, using HTTP PUT with Postman, it gives me the following error, with Status 403:

<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature 
 you provided. Check your Google secret key and signing method.</Message>
<StringToSign>PUT

 multipart/form-data; boundary=------------------------- 
 -025804137217704409263172
 1553069420
 /roshanbucket/pi.jpg</StringToSign>
 </Error>

I have no idea what's causing this. Some help would be really appreciated.

Roshan Upreti · Accepted Answer · 2019-03-22T00:23:40.317

After a while of struggle, finally managed to get it running. Turns out, first I need to generate a signed URL, the equivalent of

    gsutil signurl -c 'Content-Type' \
   -m RESUMABLE /path/to/your/json_cert_file.json \
    gs://your_bucket/file.txt

Then using that signed URL, send an empty POST request with Content-Type and x-goog-resumable:start headers, the equivalent of

    curl -v -X 'POST' \
   -H 'content-type: text/plain' \
   -H 'x-goog-resumable:start'  \
   -d '' '<signedURL>'

A successful POST will return status 201 with Location header with the actual location where you can upload the file using HTTP PUT.

Below is the Java class I wrote to finish this, with the help of this article

    import com.google.api.client.util.Base64;
    import com.google.auth.oauth2.ServiceAccountCredentials;
    import java.io.DataOutputStream;
    import java.io.File;
    import java.io.FileInputStream;
    import java.io.IOException;
    import java.io.InputStream;
    import java.net.HttpURLConnection;
    import java.net.URL;
    import java.net.URLEncoder;
    import javax.ws.rs.client.Client;
    import javax.ws.rs.client.ClientBuilder;
    import javax.ws.rs.client.Entity;
    import javax.ws.rs.client.ResponseProcessingException;
    import javax.ws.rs.core.Response;
    import uploader.Utilities;

    public class Uploader {

      private ServiceAccountCredentials creds;    // Service Account Credentials
      private String saEmail;                     // Service Account email

      public Uploader() {
        /* Initialize credentials and service account email*/
        try (InputStream inputStream = new FileInputStream("C:/cred/Key.json")) {
          this.creds = ServiceAccountCredentials.fromStream(
              inputStream);
        } catch (IOException e) {
          e.printStackTrace();
        }
        this.saEmail = "service account email";
      }

      /* Sign and return the URL for POST, using credentials from above*/
      private String getSignedUrl(String bucketName, String objectName, String mimeType) {
        String signed_url = null;
        try {
          String verb = "POST";
          long expiration = System.currentTimeMillis() / 1000 + 60;
          String Canonicalized_Extension_Headers = "x-goog-resumable:start";
          String content_type = mimeType;

          byte[] sr = creds.sign(
              (verb + "\n\n" + content_type + "\n" + expiration + "\n" + Canonicalized_Extension_Headers
                  +
                  "\n" + "/" + bucketName + "/" + objectName).getBytes());
          String url_signature = new String(Base64.encodeBase64(sr));
          signed_url = "https://storage.googleapis.com/" + bucketName + "/" + objectName +
              "?GoogleAccessId=" + saEmail +
              "&Expires=" + expiration +
              "&Signature=" + URLEncoder.encode(url_signature, "UTF-8");
        } catch (Exception ex) {
          ex.printStackTrace();
        }
        return signed_url;
      }


      /* Send POST request to the signed URL using custom headers and an empty body, which returns the actual upload location */
      public String getLocation(String bucketName, String objectName, String mimeType)
          throws IOException {
        URL myURL = new URL(getSignedUrl(bucketName, objectName, mimeType));
        HttpURLConnection myURLConnection = (HttpURLConnection) myURL.openConnection();
        myURLConnection.setRequestMethod("POST");
        myURLConnection.setRequestProperty("Content-Type", mimeType);
        myURLConnection.setRequestProperty("x-goog-resumable", "start");
        // Send post request
        myURLConnection.setDoOutput(true);
        DataOutputStream wr = new DataOutputStream(myURLConnection.getOutputStream());
        wr.flush();
        wr.close();
        int responseCode = myURLConnection.getResponseCode();
        if (responseCode != 201) {
          System.out.println("Request Failed");
        }
        return myURLConnection.getHeaderField("Location");            
      }

      /* Do the actual upload and return the PUT Response*/
     public Response doUpload(String url, InputStream inputStream, String mimeType) {
        Response response = null;
        Client client = ClientBuilder.newClient();
        try {
          response = client.target(url)
              .request()
              .put(Entity.entity(inputStream, mimeType));
          if (response.getStatus() != 200) {
            System.out.println("Request failed with " + response.getStatus());
          }
        } catch (ResponseProcessingException e) {
          e.printStackTrace();
        }
        return response;
      }   

    }

Now, simply call it in the main method

public static void main(String[] args) throws Exception {
Uploader uploader = new Uploader();    
String filePath = "file/path";
File file = new File(filePath);
byte[] bytes = Utilities.fileToByteArray(file); // convert file to bytes
String mimeType = Utilities.getMimeType(bytes); // bytes from above used with tika
String url = uploader.getLocation("bucket", file.getName(), mimeType);
Response r = uploader.doUpload(url, new FileInputStream(file), mimeType);
System.out.println("Response : " + r.getStatus());
System.out.println(r.getHeaders());
}

Hope this helps someone! This method doesn't require sending POST request with Jwt in Authorization Bearer.

score 1 · Answer 2 · answered Dec 23 '19 at 21:22

If you are using SpringBoot and RestTemplate then this should work:

  // IMPORTS

  import org.springframework.web.client.RestTemplate;

  import org.springframework.core.io.FileSystemResource;
  import org.springframework.http.HttpEntity;
  import org.springframework.http.HttpHeaders;
  import org.springframework.http.HttpMethod;

  import java.io.File;

...

  // USE THIS TO UPLOAD YOUR FILE

  String signedurl = "https://storage.googleapis.com/..."; // Your signed URL
  File file = new File(); // Your file
  FileSystemResource resource = new FileSystemResource(file);
  HttpHeaders headers = new HttpHeaders();
  headers.add("Content-Type", "plain/text"); // Should be the same MediaType used to create the signedUrl
  HttpEntity<FileSystemResource> requestEntity = new HttpEntity<>(resource, headers);

  restTemplate.exchange(signedUrl, HttpMethod.PUT, requestEntity, String.class); // PUT is the HTTP method used to create the signedUrl

...

Uploading to Google Cloud using a signed URL

2 Answers2

Linked