Generate gcs signed URL to upload large files

Question

I have been trying to upload large files by generating a signed URL. Here is the documentation I was following to generate the signed URL: https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#code-samples

It was working fine for files in 100's of MB's but as soon as the file size went up to 1 GB, the curl command started timing out even after increasing the expiration time. I tried looking at the answer here: https://stackoverflow.com/a/63789297/7466551, but I am still unable to get the URL working to upload the URL.

I am using this command to upload the file: curl -X POST -H 'x-goog-resumable: start' --upload-file file-name 'pre_signed_google_url'. I am adding the 'x-goog-resumable: start' header as I am having "x-goog-resumable", "start" header as a part of my code to generate the URL.

Can someone please let me know if I need to do any additional thing to generate the URL to upload large files?

`x-goog-resumable:start` indicates that you're trying to initialize a resumable upload. Such a call should be a POST, not a put. Take a look at this guide: https://cloud.google.com/storage/docs/performing-resumable-uploads#xml-api — Brandon Yarbrough, Feb 23 '21 at 19:11
Thank you for pointing the bug in my command. I was able to get the link working after changing my curl command to POST. I am updating my question. Also, I found this article: https://medium.com/google-cloud/google-cloud-storage-signedurl-resumable-upload-with-curl-74f99e41f0a2 that explains creating signed URL for resumable uploads. I was able to use a combination of the StackOverflow link that I provided in the question and the medium article to come up with the solution I was looking for. I will post my answer shortly. — Jaiganesh Prabhakaran, Feb 25 '21 at 00:50

Jaiganesh Prabhakaran · Answer 1 · 2021-02-25T01:03:36.847

Answering my own question as I had to use two separate sources to reach a solution.

On top of the java code here: https://stackoverflow.com/a/63789297/7466551, I referred to the medium article here: https://medium.com/google-cloud/google-cloud-storage-signedurl-resumable-upload-with-curl-74f99e41f0a2

So you need the following additional lines of code on top of the StackOverflow answer to get a signed URL for resumable uploads:

// Open a HTTP connection and add header
URL obj = new URL(url.toString());
HttpURLConnection connection = (HttpURLConnection) obj.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("x-goog-resumable", "start");
connection.setDoOutput(true);

// Connect to the URL and post headers
DataOutputStream writer = new DataOutputStream(
        connection.getOutputStream());
writer.writeBytes("");
writer.flush();
writer.close();

// Checking the responseCode to
if (connection.getResponseCode() == connection.HTTP_CREATED) {
    connection.disconnect();
    System.out.println("Location: " + connection.getHeaderField("Location"));
}
else {
    // Throw error
    BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
    String inputLine;
    StringBuffer response = new StringBuffer();
    while ((inputLine = reader.readLine()) != null) {
        response.append(inputLine);
    }
    reader.close();
    String errorMessage = response.toString();
    connection.disconnect();
    throw new IOException(errorMessage);

}

Generate gcs signed URL to upload large files

1 Answers1