Browser Javascript: Compress Json to gzip and upload to S3 presigned URL

Question

Any advice would be appreciated. I've got a json variable in my web application that I'd like to gzip and upload to S3 through a presigned URL.

I'm able to upload JSON successfully, but I fail to gzip the JSON and then upload it.

The three separate different ways I've tried to build the gzipped json is:

// example json
const someJson = { testOne: 'a', testTwo: 'b' };

// Attempt one
const stringUtf16 = JSON.stringify(someJson);
const resultAsBinString = pako.gzip(stringUtf16);

// Attempt two
const stringUtf16 = JSON.stringify(someJson);
const resultAsBinString = pako.gzip(stringUtf16, { to: 'string' });

// Attempt three
const stringUtf16ThatWeNeedInUtf8 = JSON.stringify(someJson);
const stringUtf8 = unescape(encodeURIComponent(stringUtf16ThatWeNeedInUtf8));
const resultAsBinString = pako.gzip(stringUtf8);

For each attempt, I uploaded the resultAsBinString through Angular's HTTP client, with the headers Content-Type: 'application/x-gzip' and Content-Encoding: 'gzip'

But when (and if, oftentimes it gives a network error) the file is afterwards downloaded from S3, when trying to unzip with gzip or gunzip in the terminal, an error message is given: 'not in gzip format'

Sources I've tried to follow:

https://github.com/nodeca/pako/issues/55 https://github.com/nodeca/pako/blob/master/examples/browser.html

score 5 · Answer 1 · answered Jul 26 '19 at 20:35

Setting Content-Encoding: gzip is not correct, if you expect the payload to remain gzipped after the download. This is only used when you want the browser to transparently decode the gzip encoding -- such as when serving gzipped HTML, JavaScript, CSS, etc.

If Content-Encoding: gzip is used, then Content-Type should be set to match the actual payload, such as Content-Type: application/json.

If Content-Type: application/x-gzip is used, the Content-Encoding should not be used, unless you're using a different sort of compression to re-compress the gzip payload (unlikely).

Content-Type: application/x-gzip combined with Content-Encoding: gzip means you have wrapped a gzipped file inside another layer of gzip compression and you want the outer layer removed by the browser, which is not something you would ever do in practice.

score 5 · Accepted Answer · answered Jul 29 '19 at 10:46

The following process worked for me:

Generate the presigned URL with Content-Type: 'application/json'. The provided filename should include the .gz at the end. In the returned presigned URL, scanning through the URL should verify the Content Type is application/json.

Because I'm certain my JSON contains no strings that would break the conversion to UTF-8, I then do the following (code in Angular, but it conveys the structure):

const headers = new HttpHeaders({
    'Content-Type': 'application/json',
    'Content-Encoding': 'gzip'
});  //1
const httpOptions = {
    headers: headers
};
const str = JSON.stringify(geoJson); //2
const utf8Data = unescape(encodeURIComponent(str)); //3
const geoJsonGz = pako.gzip(utf8Data); //4
const gzippedBlob = new Blob([geoJsonGz]); //5
upload = this.httpClient.put(presignedUploadUrl, gzippedBlob, httpOptions); //6

Steps followed in the code:

The Content Type header is application/json, and the Content-Encoding is gzip.
Stringify the JSON
Convert the string to UTF-8
Gzip the string
Create a file from the zipped data
Upload the file to the presigned URL

You can then download the gzipped file from S3 (it should automatically be unzipped by the browser) and open it to verify that it contains the same results.

Browser Javascript: Compress Json to gzip and upload to S3 presigned URL

2 Answers2