1

I have a motion detection script that uses the Picamera library. I use the PicameraCircularIO stream object, that inherits from the CircularIO class (still a class form the Picamera library), which inherits from the native Python BytesIO class.

The problem is that when recording into the circular stream and then writing the content of the stream on the disk, the memory does not get cleared.

After the stream has been written on the disk, this is what I do:

stream.seek(0)
stream.truncate()

I have also tried calling stream.flush() and/or closing the stream: stream.close(), but it did not help.

I have also tried removing all references to the stream for it to be garbage collected (even though it makes no sense to do that, since truncating the stream to the size 0 should normally do the job...). The stream is in an array, so I did del streams[index]. It did not work, so I also tried calling the garbage collector manually: gc.collect(), with no effect.

It is only the bytes that are being writtin in the stream that are being counted in the memory, not the stream size. So for example, if 2125000 bytes are written into the stream and then I truncate it, there will still be 2125000 bytes in the memory.

Any tip/trick/answer is appreciated.

Cheers!

Maxime Dupré
  • 5,319
  • 7
  • 38
  • 72
  • 1
    I have no idea about this picamera library. How are you measuring your memory size (i.e. how do you know you are not freeing memory)? Is the problem your process is still using memory? If so, see http://stackoverflow.com/questions/15455048/releasing-memory-in-python You can spawn a subprocess to read the stream and have the subprocess exit when you are done, which would release the memory. – RobertB Nov 02 '15 at 19:31
  • I am monitoring the memory with the `top` command while my script is being executed. The link you posted might be useful, I will look at that soon! – Maxime Dupré Nov 02 '15 at 19:35
  • 1
    Yes, I believe, 'gc.collect' will flag memory as reusable within the process, but won't necessarily let it go. Your process may maintain it's size at the "high water mark" of memory usage. Thus the idea to launch a subprocess and have it exit when done. The os will always take back the memory this way. – RobertB Nov 02 '15 at 19:40
  • That was a very good read! If I understand correctly, the memory % that is shown in the `top` command isn't necessarily the memory that is being used, but rather the memory that is being reserved by the script (a.k.a. the water mark). Then I have no problem if the memory is actually free. That explains why when I empty a stream, and start to fill another one, the memory % doesn't go up, because it has to trespass the water mark for the % to increase. – Maxime Dupré Nov 03 '15 at 01:48
  • 1
    Yeah, if you start to fill another stream, your process should not grow in size, it should start reusing the same space that it already has been allocated. Unless you get some sort of internal memory fragmentation or other sort of memory leak, you should be ok. – RobertB Nov 03 '15 at 01:51
  • Cool thanks! Since the python script is the "only process" running on the machine, I have no problem with the memory being all reserved by that script. Also can you post that as an answer? – Maxime Dupré Nov 03 '15 at 01:53

1 Answers1

1

gc.collect is flagging the memory as unused. As your program requires more memory, it should reuse that memory. However, it may not release that memory back to the OS, so the OS will continue to report the memory as used. Generally speaking, python processes tend to hold on to their memory and not release it back, so their total memory footprint will represent the 'high water mark' of their memory usage.

In your case, this should be ok since you are releasing the stream. If you start to use another stream, it should reuse the same space and your 'top' report should not show an increase in size.

See Releasing memory in Python for a more detailed explanation.

Community
  • 1
  • 1
RobertB
  • 1,879
  • 10
  • 17