How to remove keys from TypedArray in JSON

Question

I'm trying to store large, typed-arrays from Javascript into JSON-Files.

After stringifying one typed-array, I stored the JSON string in a new File using nodes' file stream system.

So far so good, but, once I checked the newly created file, I noticed that the size of this file is huge. After looking through the actual text stored in the file I noticed quite a lot of potential for improving for reducing the files size.

This exemplaric structure illustrates the structure of the array in the actual JSON string:

{"data": [{"my_key": "my_val1"},{"my_key": "my_val2"},{"my_key": "my_val3"},{"my_key": "my_val4"},{"my_key": "my_val5"},{"my_key": "my_val6"},{"my_key": "my_val7"},{"my_key": "my_val8"},{"my_key": "my_val9"}]}

Now, after googeling around a bit, I found this libary (json-compressed) which is exactly what I want to do with my array. Unfortunately this libary is made for php, I tried other libaries like JSONC and various other libaries that used some sort of compression algorithm. (LZW, DEFLATE/gzip).

I found that all libaries that use some sort of compression algorithm aren't able to provide a solution for my problem as I have to remove the keys before trying to compress the file even further.

Unfortunately, promising libaries like JSONC simply crashed my Application without throwing an error, notification or something else whenever I used their function JSONC.compress(). Their JSONC.pack() function crashes as well, but with giving away an error message. After looking through the issues related to JSONC on github I realized that this project is probably dead anyways.

Now back to my original question, is it possible to remove the keys from each value in the array? And if it is: Whats the smartest approach to do so?

To clear things up, the keys are NOT CREATED by me or PRESENT in my object! I simply want to stringify a TYPED-ARRAY but without getting a key for each value inside this array in the final JSON-STRING.

A result like this would be perfect and decrease the size of my file tremendously:

{"data": {"keys": ["my_key"], "data":["my_val1","my_val2","my_val3","my_val4","my_val5","my_val6","my_val7","my_val8","my_val9"]}}

Can't you just group this list by key? I mean, something similar to: `{data: { my_key: [my_val1, my_val2, ...]}}` — raina77ow, May 05 '17 at 15:24
Related: http://stackoverflow.com/questions/14446511/what-is-the-most-efficient-method-to-groupby-on-a-javascript-array-of-objects — raina77ow, May 05 '17 at 15:31
That doesn't seem to be related as they got objects that explicitly got created with key properties. All I got is a json STRING, I cant remove 'keys' from my object because JSON itself is creating the keys for each value in my typed-array (The key for each value is just the position where it stood in the array; from 0-n) — mxl, May 05 '17 at 15:36

score 2 · Answer 1 · answered Jun 23 '20 at 21:47

You can use a replacer function as the second argument to JSON.stringify to convert typed arrays into normal arrays. No need to modify or create objects.

(k, v) => ArrayBuffer.isView(v) ? Array.from(v) : v

const myObj = {
    floats: new Float32Array([ 0.5, 3.1, -0.2, 8.0 ]),
    uints: new Uint16Array([ 259, 2 ]),
};
return JSON.stringify(myObj, (k, v) => ArrayBuffer.isView(v) ? Array.from(v) : v, /*indent*/ 2);

{
  "floats": [
    0.5,
    3.0999999046325684,
    -0.20000000298023224,
    8
  ],
  "uints": [
    259,
    2
  ]
}

James · Accepted Answer · 2017-05-05T17:23:02.073

1

JSON.stringify treats typed arrays as objects with 0-based indices rather than arrays. Converting a typed array to a standard array using Array.from before encoding would work, for example:

var myTypedArray = new Float32Array([0.1, 323.3, 0, 2.2]);
console.log(JSON.stringify(myTypedArray)); // looks like an object
var stdArray = Array.from(myTypedArray);
console.log(JSON.stringify(stdArray)); // looks like an array

Then to convert the JSON back to a typed array:

var json = "[0.1, 323.3, 0, 2.2]";
var myTypedArray = new Float32Array(JSON.parse(json));

edited May 05 '17 at 17:23

answered May 05 '17 at 15:38

James

20,957
5
26
41

Thank you for your fast reply! Unfortunately this is not a correct representation of my problem. As I mentioned in my initial question the keys are created by JSON once the typedArray in my object is being stringified. – mxl May 05 '17 at 15:44
No its not. The Data thats causing my problem is a Float32Array inside the JS obj I'm trying to store. This typed-(Float32)Array is being equipped with keys (Where the value stood in the actual array; from 0-N) by JSON automagically once it gets stringified (and only when it gets stringified!). – mxl May 05 '17 at 15:58
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Float32Array – mxl May 05 '17 at 15:59
Hey James thanks for the hint, I wasn't aware that this is possible so easily! Your solution in combination with the usage of lzwcompress (One can find it in npm) brought the file down to a realistic size. @James: If you could update your answer to prevent misconceptions you'll get the points for solving. Thank you very much :) – mxl May 05 '17 at 16:50

score 0 · Answer 3 · answered May 05 '17 at 15:35

Working example of what you want:

var dataObject = {
    "data": [{
        "my_key": "my_val1"
    }, {
        "my_key": "my_val2"
    }, {
        "my_key": "my_val3"
    }, {
        "my_key": "my_val4"
    }, {
        "my_key": "my_val5"
    }, {
        "my_key": "my_val6"
    }, {
        "my_key": "my_val7"
    }, {
        "my_key": "my_val8"
    }, {
        "my_key": "my_val 9 "
    }]
};

var newDataObject = { "data": {} };
var data = dataObject.data;

for (var i = 0; i < data.length; i++) {
    for (key in data[i]) {
        if (!newDataObject.data.hasOwnProperty(key)) {
            newDataObject.data[key] = [];
        }
        newDataObject.data[key].push(data[i][key]);
    }
}

console.log("New data object is ", JSON.stringify(newDataObject, null, 4));

Thank you for your fast reply! Unfortunately this is not a correct representation of my problem. As I mentioned in my initial question the keys are created by JSON once the typedArray in my object is being stringified. — mxl, May 05 '17 at 15:44

How to remove keys from TypedArray in JSON

3 Answers3