0

I'm posting huge amounts of data from my webworker where I parse many 100mbs of files.

for(let y = 0; y < nrOfFiles; y++) {
    let items = parsedFiles[y].items;
    let nrOfItems = parsedFiles[y].items.length;
    for(let x = 0; x < nrOfItems; x++){
      postMessage({
        aTopic: 'file',
        fileIdx: y,
        item: JSON.stringify(parsedFiles[y].items[x])
      });
    }

On the receiving end:

  worker.onmessage = function (e) {
      if (e.data.aTopic === 'file') {
        parsedFiles[e.data.fileIdx].items.push(JSON.parse(e.data.item))
      }
  }

The tab in Chrome crashes when the data gets to big. Is there any way I can do this more effective?

Joe
  • 4,274
  • 32
  • 95
  • 175
  • Post the data incrementally or transfer the object as an `ArrayBuffer`. – guest271314 Feb 01 '17 at 17:17
  • If by “Chrome crashes” you mean the chrome process quits and you have to restart the browser, then that is a bug in Chrome that should be reported at https://bugs.chromium.org/p/chromium/issues/list. Nothing that you do from a Web document/application should ever be able to crash the browser. Among other reasons because if something can crash the browser, then it could be exploited by malicious sites to intentionally crash users’s browsers. – sideshowbarker Feb 01 '17 at 17:30
  • @sideshowbarker Perhaps OP means the `tab` crashes, which is possible [Web workers inside promise causing crash](http://stackoverflow.com/questions/40066129/web-workers-inside-promise-causing-crash). It could be that OP is calling `postMessage` from within a `for` loop. Though presently not enough information at Question to reproduce tab crash. – guest271314 Feb 01 '17 at 17:45
  • @sideshowbarker, No it's not crashing. It just says something went wrong. Don't know what to call it? – Joe Feb 01 '17 at 17:50
  • @guest271314. Yes I call it from a for-loop – Joe Feb 01 '17 at 17:50
  • @guest271314. JSON.stringify crashes if I do it in a single post. – Joe Feb 01 '17 at 17:51
  • @PerStröm Again, why do you not post the data incrementally? – guest271314 Feb 01 '17 at 17:55
  • Can you create a plnkr http://plnkr.co to reproduce issue? – guest271314 Feb 01 '17 at 18:04
  • @guest271314, Post incrementally? Not sure what you mean. Do you have an example? – Joe Feb 01 '17 at 18:08
  • @PerStröm There are several possible approaches [jQuery - Can threads/asynchronous be done?](http://stackoverflow.com/questions/26068821/jquery-can-threads-asynchronous-be-done/), [createImageBitmap alternative on Safari](http://stackoverflow.com/questions/40094245/createimagebitmap-alternative-on-safari), [Chrome memory issue - File API + AngularJS](http://stackoverflow.com/questions/41440235/chrome-memory-issue-file-api-angularjs). Can you reproduce `tab` crash at plnkr http://plnkr.co? – guest271314 Feb 01 '17 at 18:15
  • Was to tricky to to a web worker on plunkr. – Joe Feb 01 '17 at 18:36
  • @PerStröm _"Was to tricky to to a web worker on plunkr."_ Not sure what you mean? Should be same as how you are currently trying. See http://stackoverflow.com/help/mcve – guest271314 Feb 01 '17 at 22:21
  • Generally speaking `JSON.stringify(...)` is going to perform terribly on very large data sets. In my experience I've reworked the logic so that the "large data" is never present on the main thread... only the final result. If that is infeasible here I'd agree with guest271314: use Array Buffers instead of Strings as the structure to transfer data to the worker(s). – Peter Wagener Feb 02 '17 at 17:05

0 Answers0