0

I have a 20MB .json.gz file, when uncompressed it becomes 280MB, I'd like to obtain the corresponding Javascript object in my web page so that I can do some stuffs. Unfortunately it is now clear that standard methods are buggy when reaching the 256MB limit.

There are two versions of the file, v3 is 20MB and v2 is 16MB (uncompressed 280MB and 230MB)

For the v2 file a solution worked: using jQuery

  $.getJSON( "./data_package2.json.gz" , function( res ){    /* res contains the parsed object */ });

With the v3 file it now fails somewhere during parsing (hard to debug jQuery's code so I can't say more, also the error message depends on jQuery's version).

Concretely in this web page the v2 button works fine but not the v3.

I tried loading the compressed json files in Python and both of them work

    pip install compress_json
    python

        import compress_json
        D1 = compress_json.load("data_package3.json.gz")
        D1["case_data"][1]
            // it works fine ... even if it is using 800MB of RAM..

I'd like some help to understand what fails in jQuery's code and eventually to find a javascript zlib/JSON-parser code that will work on the v3 file.

reuns
  • 854
  • 10
  • 14
  • I just realized v2 works on chrome (win32) but not on firefox, jQuery gets an exception `SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data` – reuns Feb 01 '21 at 19:28

3 Answers3

2

I was able to parse your large test file using Firefox Developer Edition and my own WASM GZip library wasm-gzip:

import init, { decompressStringGzip } from "../wasm_gzip.js";

init().then(() => {
    fetch("./data_package3.json.gz")
        .then((response) => response.arrayBuffer())
        .then((buffer) => {
            let arr = new Uint8Array(buffer);
            console.log(arr);
            let decompressed = decompressStringGzip(arr);
            console.log(decompressed);
            let obj = JSON.parse(decompressed);
            console.log(obj);
        });
});

Firefox Developer Edition

Firefox Console Screenshot

Google Chrome

Chrome Console Screenshot

  • My x64-based PC has 16GiB RAM.
  • Google Chrome: Version 88.0.4324.104 (Official Build) (64-bit)
  • Firefox (Developer): 86.0b4 (64-bit)

I would recommend decompressing your file using a WebWorker, because the page is unresponsive while all the parsing and decompressing is done.

B. Colin Tim
  • 293
  • 5
  • 13
  • Thank you, I think I created the [correct environment there](http://www.paris8.free.fr/ncov/browser.html), both json fail on chrome and firefox when creating the big string `Uncaught (in promise) RuntimeError: unreachable at alloc::fmt::format::hb786be8f7eacdff5 at libflate::gzip::Decoder::new::hbb78d053d8cca61a at wasm_gzip::gzip_decompress::h450743b1eb6d1c76 at decompressStringGzip` there are some chances V8 doesn't have the same string limit size on my 32bit OS and yours. I'd like the JSON to load on standard OS & browser. – reuns Feb 01 '21 at 21:12
  • This must be a hardware-dependent bug. I have now tried it with Google Chrome and it worked on my system. I will update my answer with my relevant system details. Maybe JavaScript just doesn't have enaugh resources to allocate such a huge amount of data. – B. Colin Tim Feb 02 '21 at 13:31
1

There is a code that works

<script src="uint8array-json-parser.min.js"></script>
<script>
var result;
function data() {   
    var req = new XMLHttpRequest(),req2 = new XMLHttpRequest();
    req.responseType = "arraybuffer"; 
    req.onload = function(oEvent) {
      var arrayBuffer = req.response;
      var data = new Uint8Array(arrayBuffer);
      result = JSON_parse(data); // from uint8array-json-parser.min.js
    };
    req.open("GET", "big.bin", true);
    req.send();
}
</script>

where uint8array-json-parser.min.js comes from this package, you can find it there.

big.bin is a json.gz file, which means you need to modify the .htaccess this way (when the file is on a Apache server you have access to):

<FilesMatch ".*\.bin$">
    <IfModule headers_module> 
      #some servers might need mod_headers.c instead of headers_module, and some others might refuse such a modification. Also sometimes there is a line deny override in the root folder's .htaccess
        Header set Content-Encoding "gzip"
        Header set Content-Type "binary/octet-stream"
    </IfModule>
</FilesMatch>

What is important is the Content-Encoding: gzip in the header telling the browser to decompress the downloaded file before serving it in req.response in javascript.

Be careful that chrome is adding Content-Encoding: gzip when it sees a .json.gz file whereas firefox is not.

The main problem with this approach is that uint8array-json-parser.min.js is a bit slower than JSON.parse. I would be interested to hear why JSON.parse doesn't accept a UInt8array (in V8 C++ it will just be a pointer cast), maybe JSON.parse prefers a string because it is immutable?

It is also possible to decompress the json.gz in javascript (with say pako.js) but again this is slower than the native implemention (we don't have access to, again I would be interested to hear why).

reuns
  • 854
  • 10
  • 14
0

A partial answer:

Splitting the big json in two files (10MB compressed each, uncompressed 150MB and 130MB) and merging the parsed objects in javascript works fine as shown there

So this is likely a problem of largest string size or similar.

I'm using windows 7 32bit and chrome, this might have an impact.

Should I try to open a ticket on jQuery since they know better at which step the big file is likely to fail?

reuns
  • 854
  • 10
  • 14