5

I'm trying to simulate a resumable-upload to a GCS signed-url using Postman to do two PUT to see how it works.

After reference several posts:

Can I do a resumable upload with a Google Cloud Storage signed URL?

Uploading to Google Cloud using a signed URL

How to use gcs-resumable-upload with signed url

I finally made it and successfully get back the 200 OK, but the file is corrupted. Here are the steps:

0). POST to our server (running NodeJS 8.11) to get a bucket signed-url with action=="resumable". Server code is:

      const file = bucket.file(fileName);
      const config = {
        action: 'resumable',
        version: 'v4',
        expires,
        contentType
      };
      file.getSignedUrl( config, (error, url) => {
        if (error) return reject(error);
        resolve(url);
      });

which will get back the signed-url as:

https://storage.googleapis.com/<bucket_name/<path_to_file>?GoogleAccessId=<service_account>&Expires=<expiry_time>&Signature=<signature>

1). POST to the signed url with header { 'x-goog-resumable': 'start' } and an empty body. I got back HTTP 201 created, and the "Location" header with value same as the signed-url, plus the &upload_id=AEnB2UqIJL_jf..... at the end, also the "X-GUploader-UploadID" header with the upload_id.

2). To simulate 2 uploads for a single file size of 730417, first PUT to the "Location" returned from (1), with Headers:

Content-Length: 262144
Content-Range: bytes 0-262143/730417
Content-Type: application/pdf

and in Postman select body as "binary" and choose the source file. I received the response of HTTP 308 (Resume Incomplete) and in the response header:

Content-Type: text/plain; charset=utf-8
Range: bytes=0-262143
X-Range-MD5: ccafa0beacd8c342bb460b95249114dc
X-GUploader-UploadID: AEnB2UqIJL_jf.....

3). To check the current upload status, following step (2) I do a PUT to same "Location" url, with empty body and header:

Content-Length: 0
Content-Range: bytes */730417

I got back same HTTP 308 (Resume Incomplete) as in (2), and response header:

Content-Type: text/plain; charset=utf-8
X-GUploader-UploadID: (same_upload_id)
Range: bytes=0-262143
X-Range-MD5: ccafa0beacd8c342bb460b95249114dc

and at this moment, I still don't see the uploaded-file in my bucket yet from GCS bucket console.

4). So far so good, so I finish it up with 2nd part of PUT to same "Location" url with following header and select the body as "binary" and choose the same source file:

Content-Length: 468273
Content-Range: bytes 262144-730416/730417
Content-Type: application/pdf

and I got back HTTP 200 OK with response header:

x-goog-generation: 1568787913806411
x-goog-stored-content-length: 730417
ETag: "018677cb75c5a307217eec21fa18ee91"
Content-Length: 0
Content-Type: text/html; charset=UTF-8

and I can see the uploaded-file appeared in the GCS console.

5). The problem now is that: the file downloaded from GCS has the same byte-count as the source file, but MD5 checksum mismatch with the source file. The source file is a 3 page PDF file, and the uploaded file, after downloaded it, has only the last page.

What could go wrong ? ... did I do anything wrong in Postman steps ? ... The steps seems to follow the steps here https://cloud.google.com/storage/docs/xml-api/resumable-upload properly, but the corrupted content is quite mystery to me, and I'm not sure where to troubleshoot this ... maybe using another client to try the resumable upload to signed-url ? ... any suggestion is very much appreciated, Thanks !

Roger PP.
  • 73
  • 1
  • 7
  • Very interesting what you're trying to do. Can you please add the full response you get from step 1? – Kevin Quinzel Oct 07 '19 at 22:55
  • @Kevin sure, the response has status 201, body as "1" and the following response headers: `Content-Type: text/plain; charset=utf-8` `X-GUploader-UploadID: AEnB2UrF...(an_upload_id)` `Location: https://storage.googleapis.com/?GoogleAccessId=&Expires=&Signature=&upload_id=AEnB2UqIJL_jf..... ` `Content-Length: 0` `Date: Tue, 08 Oct 2019 03:48:34 GMT` `Server: UploadServer` `Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000` – Roger PP. Oct 08 '19 at 03:57
  • I was following the steps you followed in the doc and the ones in your Question. I noticed that in your Step #1, you might have forgotten to add the Content-Lenght:0 header. But this doesn't seem to have effect in the response you're getting. However, what caught my attention is the Content-Type. I saw that you're receiving a text/plain in the response but then you manage it later with the Content-Type header pdf/application. I think that it could lead to the corruption you're seeing... But I'm not sure at all... – Kevin Quinzel Oct 08 '19 at 18:18
  • 1
    @Kevin, I need to send the `Content-Type: application/pdf` header in step 1, 2 and 4 otherwise GCS will complain signature not-match (I've added the content-type when generating the signedUrl). The "Content-Type: text/plain" is the response header that I got back from GCS, which tells me how to interpret the response body, and, in all steps 1-4, I got "Content-Length: 0" and no body at all in response. One-way is my request-header which is PDF file, the other-way is the response-header from GCS, which I think they're correct. – Roger PP. Oct 09 '19 at 07:09

0 Answers0