1

I have a method that runs every 2 seconds to capture a video stream to canvas and write it to file:

  function capture(streamName, callback) {
    var buffer,
      dataURL,
      dataSplit,
      _ctx;

    _ctx = _canvas[streamName].getContext('2d');
    _ctx.drawImage(_video[streamName], 0, 0);
    dataURL = _canvas[streamName].toDataURL('image/png');
    dataSplit = dataURL.split(",")[1];
    buffer = new Buffer(dataSplit, 'base64');

    fs.writeFileSync(directory + streamName + '.png', buffer);
  }

  setInterval(function() {
    // Called from here
    captureState.capture(activeScreens[currentScreenIndex]);
    gameState.pollForState(processId, activeScreens[currentScreenIndex], function() {
      // do things...
    });
  }, 2000);

Assuming _video[streamName] exists as a running <video> and _canvas[streamName] exists as a <canvas>. The method works, it just causes a memory leak.

The issue:

Garbage collection can't keep up with the amount of memory the method uses, memory leak ensues.

I have narrowed it down to this line:

buffer = new Buffer(dataSplit, 'base64');

If I comment that out, there is some accumulation of memory (~100MB) but it drops back down every 30s or so.

What I've tried:

Some posts suggested buffer = null; to remove the reference and mark for garbage collection, but that hasn't changed anything.

Any suggestions?

Timeline: https://i.stack.imgur.com/DOOpS.png https://i.stack.imgur.com/3NcXu.png

Allocation Profile: https://www.dropbox.com/s/zfezp46um6kin7g/Heap-20160929T140250.heaptimeline?dl=0

Just to quantify. After about 30 minutes of run time it sits at 2 GB memory used. This is an Electron (chromium / desktop) app.

SOLVED Pre-allocating the buffer is what fixed it. This means that in addition to scoping buffer outside of the function, you need to reuse the created buffer with buffer.write. In order to keep proper headers, make sure that you use the encoded parameter of buffer.write.

Matt
  • 5,547
  • 23
  • 82
  • 121
  • Did you try doing a Timeline recording of your code using the Chrome DevTools? This will help you see if you **actually** have a memory leak. Can you post an image of the timeline? Here's an example of how to do this: https://youtu.be/LaxbdIyBkL0?t=236. – Corina Sep 29 '16 at 06:47
  • Added a few things. Let me know if I hit what you were looking for. – Matt Sep 29 '16 at 20:14
  • You may need to destroy the Buffer object by using `delete buffer;` after `fs.WriteFile`. This can delete the reference to object `buffer`, so GC can collect it. – JQian Sep 30 '16 at 23:39
  • 2
    Delete is supposed to remove properties from objects and not to delete variables: http://stackoverflow.com/questions/16963066/how-to-delete-a-variable the only thing I can find to "delete" it is to `null` or `undefined` it. – Matt Oct 03 '16 at 03:42
  • I did try attaching the buffer to an attribute: `var buffers = {}; buffers.active = new Buffer(/* things */);` but that didn't work. Neither did `null` or `undefined` – Matt Oct 03 '16 at 04:04
  • Some ideas...If you know for sure that the processing takes less than 2 seconds, you could define `buffer` outside of `capture`, so that the same variable would be reused. You could also try `fs.writeFileSync`. – ConnorsFan Oct 04 '16 at 02:32
  • Tried both of those things, still no change. – Matt Oct 04 '16 at 04:34
  • Have you tried streaming to file? See [stream-handbook](https://github.com/substack/stream-handbook#stream-handbookk) – guest271314 Oct 05 '16 at 03:15
  • 3
    From the description of the Nodejs for Buffer it seems that buffer is not in V8 heap and is not a subject to GC. It does seem a bit strange, however, that you allocate a new buffer every 2 seconds. Perhaps you should create a pool of buffers at the start and, reuse them, limit the number of connections based on your resources. – Vladimir M Oct 05 '16 at 07:01
  • Tell me how to test this. Without testing this question is unanswerable, the problem here isn't obvious. – Tomáš Zato Oct 05 '16 at 08:56
  • Similar to the suggestion from @VladimirM, you could try to reuse the buffer instead of allocating a new one. – Whymarrh Oct 05 '16 at 14:15
  • @TomášZato use Electron to capture screen https://github.com/electron/electron/blob/master/docs/api/desktop-capturer.md and use the stream in a ` – Matt Oct 05 '16 at 23:33
  • @Whymarrh I've tried scoping the `var buffer` outside with no change. – Matt Oct 05 '16 at 23:34
  • @guest271314 haven't tried that, I'll give it a shot. – Matt Oct 05 '16 at 23:35
  • @VladimirM creating a pool of buffers would be the same as scoping `var buffer` outside of the function and reusing it wouldn't it? – Matt Oct 05 '16 at 23:35
  • @Matt yes. you scope it outside of the function and allocate it only once, at the beginning of the application or whenever it is needed first time. And you have a limited amount of those buffers. – Vladimir M Oct 05 '16 at 23:57
  • @Matt Are you trying to capture still images from playing ` – guest271314 Oct 06 '16 at 00:20
  • @guest271314 yes that's correct. – Matt Oct 06 '16 at 01:26
  • @VladimirM I have tried that already, unfortunately it didn't work. I'll do some logging to confirm I didn't mess that up. – Matt Oct 06 '16 at 01:27
  • @Matt In what way it didn't work? anyways, I've posted an answer with the approximate algorithm of how buffer pool should've been used. – Vladimir M Oct 06 '16 at 06:45
  • Why don't you simply use `captureStream`, either directly from your video element, or from your canvasElement ? Also, instead of the heavy and synchronous `toDataURL`, you should at least use `toBlob`. – Kaiido Oct 06 '16 at 07:15
  • @Kaiido can you show me how to do that in an answer? – Matt Oct 06 '16 at 21:06
  • http://stackoverflow.com/questions/38924613/how-to-convert-array-of-png-image-data-into-video-file/38929517#38929517 – Kaiido Oct 07 '16 at 14:31

5 Answers5

2

Matt, I am not sure what was not working with the pre-allocated buffers, so I've posted an algorithm of how such pre-allocated buffers could be used. The key thing here is that buffers are allocated only once for that reason there should not be any memory leak.

var buffers = [];
var bsize = 10000;

// allocate buffer pool
for(var i = 0; i < 10; i++ ){
    buffers.push({free:true, buf: new Buffer(bsize)});
}

// sample method that picks one of the buffers into use
function useOneBuffer(data){
    // find a free buffer
    var theBuf;
    var i = 10;
    while((typeof theBuf==='undefined')&& i < 10){
        if(buffers[i].free){
            theBuf = buffers[i];
        }
        i++;
    }

    theBuf.free = false;
    // start doing whatever you need with the buffer, write data in needed format to it first
    // BUT do not allocate
    // also, you may want to clear-write the existing data int he buffer, just in case before reuse or after the use.
    if(typeof theBuf==='undefined'){
        // return or throw... no free buffers left for now
        return;
    }
    theBuf.buf.write(data);
    // .... continue using

    // dont forget to pass the reference to the buffers member along because 
    // when you are done, toy have to mark it as free, so that it could be used again
    // theBuf.free = true;
}

Did you try something like this? Where did it fail?

Vladimir M
  • 4,403
  • 1
  • 19
  • 24
  • I didn't even need to create a pool. I was able to write to the same buffer using `buffer.write(data);` and that seems to stop the leak. However, I'm getting an error when cropping that file using GraphicsMagick "Crop State: Command failed: [path] convert: Improper image header ([image path])". Any suggestions? – Matt Oct 06 '16 at 21:54
  • I wasn't clear-writing the buffer. I'm not sure how to do that. Could it also be that the image takes up more than the `bsize = 10000` allocated? – Matt Oct 06 '16 at 22:07
  • Hmm... I've checked the api, I was looking for something like flush in there (for clear-writing the buffer) but it seems not to be there. So Apparently its not needed. Yes 10000 bytes is a small size for an image. What is the expected size of the image - try that + some more? – Vladimir M Oct 06 '16 at 22:33
  • I tried 30MB (31457280 bytes) but it still had an issue with headers. Looking into that specific error with GraphicsMagick. Hopefully if I can get this resolved the leak doesn't come back! – Matt Oct 06 '16 at 23:13
  • Might be something to do with encoding? See here: http://stackoverflow.com/questions/8110294/nodejs-base64-image-encoding-decoding-not-quite-working – Vladimir M Oct 06 '16 at 23:18
  • It was encoding. Looks like `buffer.write` also takes a string `encoding` parameter that needed to be set to `base64`. Trying it all out to see if the memory leak is fixed now... – Matt Oct 09 '16 at 19:43
  • Once I had that in the whole thing worked. Pre-allocating a buffer and reusing the same one was the way to go. – Matt Oct 09 '16 at 20:15
0

There is no leak of buffer object in your code.

Any Buffer objects that you no longer retain a reference to in your code will be immediately available for garbage collection.

the problem caused by callback and how you use it out of capture function. notice that GC can not cleans the buffer or any other variable as long as callback is running.

fingerpich
  • 8,500
  • 2
  • 22
  • 32
  • If I used `writeFileSync` and returned from the function instead of using a callback would that help solve that? The callbacks do cycle infinitely. – Matt Oct 05 '16 at 22:26
  • Yeah I tried changing to a return instead of callback (question content updated) and it's still there. So what do you mean by change cycle method? – Matt Oct 05 '16 at 22:46
  • i thought you use callback to call next 2 second timeout. i think it is right. did you test it again? – fingerpich Oct 05 '16 at 23:06
  • Yes that was the original way it worked. Now I reworked the code to no longer need it (see question code above) and the leak is still there. – Matt Oct 05 '16 at 23:36
0

I was having a similar issue recently with a software app that uses ~500MB of data in arrayBuffer form. I thought I had a memory leak, but it turns out Chrome was trying to do optimizations on a set of large-ish ArrayBuffer's and corresponding operations (each buffer ~60mb in size and some slightly larger objects). The CPU usage appeared to never allow for GC to run, or at least that's how it appeared. I had to do two things to resolve my issues. I Have not read any specific spec for when the GC gets scheduled to prove or disprove that. What I had to do:

  1. I had to break the reference to the data in my arrayBuffers and some other large objects.
  2. I had to force Chrome to have downtime, which appeared to give it time to schedule and then run the GC.

After applying those two steps, things ran for me and were garbage collected. Unfortunately, when applying those two things independently from each other, my app kept on crashing (exploding into GB of memory used before doing so). The following would be my thoughts on what I'd try on your code.

The problem with the garbage collector is that you cannot force it to run. So you can have objects that are ready to be malloced, but for whatever reason the browser doesn't give the garbage collector opportunity. Another approach to the buffer = null would be instead to break the reference explicitly with the delete operator -- this is what I did, but in theory ... = null is equivalent. It's important to note that delete cannot be run on any variable created by the var operator. So something like the following would be my suggestion:

function capture(streamName, callback) {
    this._ctx = _canvas[streamName].getContext('2d');
    this._ctx.drawImage(_video[streamName], 0, 0);
    this.dataURL = _canvas[streamName].toDataURL('image/png');
    this.dataSplit = dataURL.split(",")[1];
    this.buffer = new Buffer(dataSplit, 'base64');

    fs.writeFileSync(directory + streamName + '.png', this.buffer);

    delete this._ctx;//because the context with the image used still exists
    delete this.dataURL;//because the data used in dataSplit exists here
    delete this.dataSplit;//because the data used in buffer exists here
    delete this.buffer;
    //again ... = null likely would work as well, I used delete
}

Second, the small break. So it appears you've got some intensive processes going on and the system cannot keep up. It's not actually hitting the 2s save mark, because it needs more than 2 seconds per save. There is always a function on the queue for executing the captureState.capture(...) method and it never has time to garbage collect. Some helpful posts on the scheduler and differences between setInterval and setTimeout:

http://javascript.info/tutorial/settimeout-setinterval

http://ejohn.org/blog/how-javascript-timers-work/

If that is for sure the case, why not use setTimeout and simple check that roughly 2 seconds (or more) time has passed and execute. In doing that check always force your code to wait a set period of time between saves. Give the browser time to schedule/run GC -- something like what follows (100 ms setTimeout in the pollForState):

var MINIMUM_DELAY_BETWEEN_SAVES = 100;
var POLLING_DELAY = 100;

//get the time in ms
var ts = Date.now();
function interValCheck(){
   //check if 2000 ms have passed
   if(Date.now()-ts > 2000){
      //reset the timestamp of the last time save was run
      ts = Date.now();
      // Called from here
      captureState.capture(activeScreens[currentScreenIndex]);
      //upon callback, force the system to take a break.
      setTimeout(function(){
        gameState.pollForState(processId, activeScreens[currentScreenIndex], function() {
          // do things...
          //and then schedule the interValCheck again, but give it some time
          //to potentially garbage collect.
          setTimeout(intervalCheck,MINIMUM_DELAY_BETWEEN_SAVES);
       });
      }
   }else{
      //reschedule check back in 1/10th of a second.
      //or after whatever may be executing next.
      setTimeout(intervalCheck,POLLING_DELAY);
   }
}

This means that a capture will happen no more than once every 2 seconds, but will also in some sense trick the browser into having the time to GC and remove any data that was left.

Last thoughts, entertaining a more traditional definition of memory leak, The candidates for a memory leak based on what I see in your code would be activeScreens, _canvas or _video which appear to be objects of some sort? Might be worthwhile to explore those if the above doesn't resolve your issue (wouldn't be able to make any assessments based on what is currently shared).

Hope that helps!

Arthur Weborg
  • 8,280
  • 5
  • 30
  • 67
  • Sorry I should've been more specific in the code I had shown. I do exactly that -- only run if completed or 2000ms pass, whichever happens last. – Matt Oct 06 '16 at 21:25
  • No dice with the `delete this.buffer;`. Thanks though! – Matt Oct 06 '16 at 21:32
  • Ah, made a slight logic error and updated. `setInterval` schedules differently than `setTimeout`. With what you just said, if your code completes after 2000ms, it's likely always crunching/operating and never giving the time to to GC. with setTimeout, as proposed, you can set a delay period for that to be at least some time (above being 100ms) for it to garbage collect/schedule garbage collect. Using setTimeout as proposed is ever so slightly different than what your code is doing. – Arthur Weborg Oct 06 '16 at 21:46
  • My apologies, my response unedited was the same as setInterval for the most part. I fixed and added some more clarity to my answer around what I meant, but failed on the first go around! and some useful links. – Arthur Weborg Oct 06 '16 at 22:02
  • 1
    Trying an extreme example of this -- adding an extra 3000ms `setTimeout` right before it does the next round of processing. Letting it run for a while, will let you know. – Matt Oct 06 '16 at 22:13
  • Thanks for your patience, hope it goes well! Would you be able to add some logs and calculate the average time it takes to run the code segment in your original setInterval code block? – Arthur Weborg Oct 06 '16 at 22:31
  • The timeout still had the leak. Just slower to get there of course. Sure I think logging run times would be helpful in general. – Matt Oct 06 '16 at 23:07
  • Thank you Matt, for checking. If buffer still is the source of the memory leak, I have a vested/shared interest in learning as to why/helping you resolve, since I use arrayBuffers (underlying memory data store of `Uint8Array`, which `Buffer` is an instance of) all the time with big data in the browser. Also, could you share some more about `_canvas` and `_video`? I'm going to spend some time tonight poking at the HeapTimeline you shared. – Arthur Weborg Oct 07 '16 at 15:42
0

I have narrowed it down to this line:

buffer = new Buffer(dataSplit, 'base64');

Short solution is not to use Buffer, as it is not necessary to write file to filesystem, where a file reference exists at base64 portion of data URI. setInterval does not appear to be cleared. You can define a reference for setInterval, then call clearInterval() at <video> ended event.

You can perform function without declaring any variables. Remove data, MIME type, and base64 portions of data URI returned by HTMLCanvasElement.prototype.toDataURL() as described at NodeJS: Saving a base64-encoded image to disk , this Answer at NodeJS write base64 image-file

  function capture(streamName, callback) {

    _canvas[streamName].getContext("2d")
    .drawImage(_video[streamName], 0, 0);

    fs.writeFileSync(directory + streamName + ".png"
      , _canvas[streamName].toDataURL("image/png").split(",")[1], "base64");
  }

  var interval = setInterval(function() {
    // Called from here
    captureState.capture(activeScreens[currentScreenIndex]);
    gameState.pollForState(processId, activeScreens[currentScreenIndex]
    , function() {
    // do things...
    });
  }, 2000);

  video[/* streamName */].addEventListener("ended", function(e) {
    clearInterval(interval);
  });
Community
  • 1
  • 1
guest271314
  • 1
  • 15
  • 104
  • 177
  • @Matt Note, you could probably substitute ` – guest271314 Oct 06 '16 at 04:26
  • @Matt do you actually ever close your file? something like fs.closeFileSync() after you are done. Perhaps fsf writer still retains a reference to the buffer so it cannot get freed. – Vladimir M Oct 06 '16 at 17:43
  • I don't see `closeFileSync` as a [method in fs](https://nodejs.org/api/fs.html). [Someone mentioned here](https://github.com/nodejs/node-v0.x-archive/issues/5596) down the page a bit that `writeFileSync` opens and closes the file. – Matt Oct 06 '16 at 21:20
  • Your method made sense to me -- avoid creating the buffer. It didn't change anything in the memory usage unfortunately. – Matt Oct 06 '16 at 21:20
  • 1
    @Matt Which portion of `javascript` at Answer is consuming memory? Have you tried substituting ` – guest271314 Oct 06 '16 at 22:20
  • It seems as though in your answer the `fs.writeFileSync` is consuming memory, so it must be within it's usage. I do however know that in my original implementation that if I create `new Buffer` and comment out `fs.writeFile` that it would still leak -- in the original `fs.writeFile` was not the cause. – Matt Oct 09 '16 at 19:31
  • I haven't tried the `progress` event. Is there any reason that would have a different impact than `setInterval()`? – Matt Oct 09 '16 at 19:31
  • @Matt See, e.g., [How JavaScript Timers Work](http://ejohn.org/blog/how-javascript-timers-work/), [Understanding timers: setTimeout and setInterval](http://javascript.info/tutorial/settimeout-setinterval) , [An alternative to Javascript's evil setInterval](http://thecodeship.com/web-development/alternative-to-javascript-evil-setinterval/) – guest271314 Oct 09 '16 at 19:40
  • @Matt See also [Media events](https://developer.mozilla.org/en-US/docs/Web/Guide/Events/Media_events) – guest271314 Oct 09 '16 at 19:51
0

In general, I would recommend using a local map of UUID / something that will allow you to control your memory when dealing with getImageData and other buffers. The UUID can be a pre-defined identifier e.g: "current-image" and "prev-image" if comparing between slides

E.g

existingBuffers: Record<string, UInt8ClampedArray> = {} 
existingBuffers[ptrUid] = ImageData.data (OR something equivalent) 

then if you want to override ("current-image") you can (overkill here):

existingBuffers[ptrUid] = new UInt8ClampedArray();
delete existingBuffers[ptrUid]

In addition, you will always be able to check your buffers and make sure they are not going out of control.

Maybe it is a bit old-school, but I found it comfortable.

matan yemini
  • 141
  • 2
  • 4