JSON.stringify is very slow for large objects

Question

I have a very big object in javascript (about 10MB).

And when I stringify it, it takes a long time, so I send it to backend and parse it to an object( actually nested objects with arrays), and that takes long time too but it's not our problem in this question.

The problem:

How can I make JSON.stringify faster, any ideas or alternatives, I need a javaScript solution, libraries I can use or ideas here.

What I've tried

I googled a lot and looks there is no better performance than JSON.stringify or my googling skills got rusty!

Result

I accept any suggestion that may solve me the long saving (sending to backend) in the request (I know its big request).

Code Sample of problem (details about problem)

Request URL:http://localhost:8081/systemName/controllerA/update.html;jsessionid=FB3848B6C0F4AD9873EA12DBE61E6008
Request Method:POST
Status Code:200 OK

Am sending a POST to backend and then in JAVA

request.getParameter("BigPostParameter")

and I read it to convert to object using

 public boolean fromJSON(String string) {
        if (string != null && !string.isEmpty()) {
            ObjectMapper json = new ObjectMapper();
            DateFormat dateFormat = new SimpleDateFormat(YYYY_MM_DD_T_HH_MM_SS_SSS_Z);
            dateFormat.setTimeZone(TimeZone.getDefault());
            json.setDateFormat(dateFormat);
            json.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
            WebObject object;
//            Logger.getLogger("JSON Tracker").log(Level.SEVERE, "Start");
            try {
                object = json.readValue(string, this.getClass());
            } catch (IOException ex) {
                Logger.getLogger(JSON_ERROR).log(Level.SEVERE, "JSON Error: {0}", ex.getMessage());
                return false;
            }
//            Logger.getLogger("JSON Tracker").log(Level.SEVERE, "END");
            return this.setThis(object);
        }
        return false;
    }

Like This

BigObject someObj = new BigObject();
someObj.fromJSON(request.getParameter("BigPostParameter"))

P.S : FYI this line object = json.readValue(string, this.getClass()); is also very very very slow.

Again to summarize

• Problem in posting time (stringify) JavaScript bottle nick.

• Another problem parsing that stringified into an object (using jackson), and mainly I have svg tags content in that stringified object as a style column, and other columns are strings, int mainly

Answer 1

As commenters said - there is no way to make parsing faster.

If the concern is that the app is blocked while it's stringifying/parsing then try to split data into separate objects, stringily them and assemble back into one object before saving on the server.

If loading time of the app is not a problem you could try to ad-hoc incremental change on top of the existing app.

• ... App loading
• Load map data
• Make full copy of the data
• ... End loading
• ... App working without changes
• ... When saving changes
• diff copy with changed data to get JSON diff
• send changes (much smaller then full data)
• ... On server
• apply JSON diff changes on the server to the full data stored on server
• save changed data

I used json-diff https://github.com/andreyvit/json-diff to calc changes, and there are few analogs.

Answer 2

Parsing is a slow process. If what you want is to POST a 10MB object, turn it into a file, a blob, or a buffer. Send that file/blob/buffer using formdata instead of application/json and application/x-www-form-urlencoded.

Reference

An example using express/multer

Answer 3

Solution

Well just as most big "repeatable" problems go, you could use async!

But wait, isn't JS still single-threaded even when it does async... yes... but you can use Service-Workers to get true async and serialize an object way faster by parallelizing the process.

General Approach

mainPage.js

//= Functions / Classes =============================================================|
// To tell JSON stringify that this is already processed, don't touch
class SerializedChunk {
  constructor(data){this.data = data}
  toJSON() {return this.data}
}

// Attach all events and props we need on workers to handle this use case
const mapCommonBindings = w => {
  w.addEventListener('message', e => w._res(e.data), false)
  w.addEventListener('error', e => w._rej(e.data), false)
  w.solve = obj => {
    w._state && await w._state.catch(_=>_) // Wait for any older tasks to complete if there is another queued
    w._state = new Promise((_res, _rej) => {
      // Give this object promise bindings that can be handled by the event bindings
      // (just make sure not to fire 2 errors or 2 messages at the same time)
      Object.assign(w, {_res, _rej})
    })
    w.postMessage(obj)
    return await w._state // Return the final output, when we get the `message` event
  }
}

//= Initialization ===================================================================|
// Let's make our 10 workers
const workers = Array(10).fill(0).map(_ => new Worker('worker.js'))
workers.forEach(mapCommonBindings)

// A helper function that schedules workers in a round-robin
workers.schedule = async task => {
  workers._c = ((workers._c || -1) + 1) % workers.length
  const worker = workers[workers._c]
  return await worker.solve(task)
}
// A helper used below that takes an object key, value pair and uses a worker to solve it
const _asyncHandleValuePair = async ([key, value]) => [key, new SerializedChunk(
  await workers.schedule(value)
)]

//= Final Function ===================================================================|
// The new function (You could improve the runtime by changing how this function schedules tasks)
// Note! This is async now, obviously
const jsonStringifyThreaded = async o => {
  const f_pairs = await Promise.all(Object.entries(o).map(_asyncHandleValuePair))

  // Take all final processed pairs, create a new object, JSON stringify top level
  final = f_pairs.reduce((o, ([key, chunk]) => (
    o[key] = chunk,  // Add current key / chunk to object
    o                // Return the object to next reduce
  ), {})             // Seed empty object that will contain all the data
  return JSON.stringify(final)
}

/* lot of other code, till the function that actually uses this code */

async function submitter() {
  // other stuff
  const payload = await jsonStringifyThreaded(input.value)
  await server.send(payload)
  console.log('Done!')
}

worker.js

self.addEventListener('message', function(e) {
  const obj = e.data
  self.postMessage(JSON.stringify(obj))
}, false)

Notes:

This works the following way:

• Creates a list of 10 workers, and adds a few methods and props to them
  • We care about async .solve(Object): String which solves our tasks using promises while masking away callback hell
• Use a new method: async jsonStringifyThreaded(Object): String which does the JSON.stringify asynchronously
  • We break the object into entries and solve each one parallelly (this can be optimized to be recursive to a certain depth, use best judgement :))
  • Processed chunks are cast into SerializedChunk which the JSON.stringify will use as is, and not try to process (since it has .toJSON())
  • Internally if the number of keys exceeds the workers, we round-robin back to the first worker and overschedule them (remember, they can handle queued tasks)

Optimizations

You may want to consider a few more things to improve performance:

• Use of Transferable Objects which will decrease the overhead of passing objects to service workers significantly
• Redesign jsonStringifyThreaded() to schedule more objects at deeper levels.

Answer 4

You can explore libraries like fast-json-stringify which use a template schema and use it while converting the json object, to boost the performance. Check the below article.

https://developpaper.com/how-to-improve-the-performance-of-json-stringify/