How to prevent memory leaks in RubyMotion when reading large files in loops

Question

RubyMotion is supposed to do automatic memory management:

RubyMotion provides automatic memory management; you do not need to reclaim unused objects.

but, when reading large files in a loop, I encounter huge memory leaks: hundreds of MB/s per seconds, exactly as if my reading buffer was never released.

The leaks mostly go away if I use release on the reading buffer on every loop. The problem is, release makes the application crash when the loop is finished.

  def readBigBinaryFile(file)
#   PURE RUBY WOULD BE
#   source=File.open(file,'r')
    source =NSFileHandle.fileHandleForReadingAtPath(file)
    buffer_size=4096
    offset=0
    size=File.size(file)
    while ( offset + buffer_size ) <= size
#      PURE RUBY WOULD BE
#      source.seek(offset) 
#      abuffer = source.read( buffer_size )
#      abuffer=abuffer.to_s

      source.seekToFileOffset(offset)
      abuffer = source.readDataOfLength(buffer_size)
      offset+=buffer_size
      @dataPointer ||= Pointer.new(:uchar,4) 
      abuffer.getBytes(@dataPointer, length: 4)
      # memory leaks very rapidly in GBs if we don't release the buffer…
      # but this relase action will make the application crash once the doSomething lookp is finished
      abuffer.release
    end
    source.closeFile
    return false
  end

The loop is:

x=0
while x < 10000
  my_scan_binary_instance=Scan_binary.new()        result=my_scan_binary_instance.readBigBinaryFile(NSBundle.mainBundle.pathForResource("sample1MBfile", ofType:"img"))
  puts result.to_s
  x+=1
end
puts "if we have used 'abuffer.release', we are going to crash now"

I tested a pure-Ruby implementation, and had no memory leak at all, and no need for the release call.

I found "How do I prevent memory leak when I load large pickle files in a for loop?" about a memory leak in a Python loop, but the accepted solution doing abuffer=nil at the beginning of the while block in readBigBinaryFile did not work.

Is this a bug in RubyMotion's automatic memory management, or is this expected? And most importantly, how can I read big files in loops without increasing the memory usage of my RubyMotion app?

I have created a gist with the working pure Ruby implementation, and a repo of a sample application reproducing the crash.

I'd recommend reading "[ask]", and especially the link to http://catb.org/esr/faqs/smart-questions.html. "[mcve]" specifies that your sample app has to be in the question _itself_. Links to other sites are prone to rot then break, making the question worthless to others in the future. — the Tin Man, May 09 '16 at 16:15
the external links are a commodity, the question itself hopefully embeds the code that makes the question complete, and useful for others, now and later — MichaelC, May 09 '16 at 17:17

score 2 · Accepted Answer · answered May 10 '16 at 01:15

Try wrapping your loop body in autorelease_pool do ... end. This should cause the autoreleased objects to be freed each loop. Assigning nil to abuffer at the end of the loop will allow the buffer memory to be freed as there will be no more references to it.

while ( offset + buffer_size ) <= size
  autorelease_pool do
    source.seekToFileOffset(offset)
    abuffer = source.readDataOfLength(buffer_size)
    offset+=buffer_size
    @dataPointer ||= Pointer.new(:uchar,4) 
    abuffer.getBytes(@dataPointer, length: 4)

    abuffer = nil
  end
end

works great! Even less memory leaking if I also put an autorelease around `my_scan_binary_instance=Scan_binary.new()`. [Link to Apple docs](https://developer.apple.com/library/ios/documentation/Cocoa/Conceptual/MemoryMgmt/Articles/mmAutoreleasePools.html) — MichaelC, May 10 '16 at 06:06

score 1 · Answer 2 · edited May 09 '16 at 16:18

1

I don't know if this will fix it for sure, but maybe try replacing

abuffer = source.readDataOfLength(buffer_size)

with

abuffer = WeakRef.new(source.readDataOfLength(buffer_size))

See "4.1. Weak References" for info on WeakRef.

Making this change would mean you would also remove the call to abuffer.release.

edited May 09 '16 at 16:18

the Tin Man

158,662
42
215
303

answered May 09 '16 at 15:44

Eric Henderson

26
4

good idea! But it dos not work. So I tried putting WeakRefs everywhere I could, and a `.weak!` at the the while loop in `doSomething`… to no avail! I also read https://spin.atomicobject.com/2013/12/09/cyclical-references-rubymotion/ – MichaelC May 09 '16 at 15:56
Welcome to SO. Please use better anchor text for links and refrain from using "Edit" or "Update". Questions and answers have revision history so we can see changes if necessary; Add changes as if you'd written them initially. Better anchor text helps keep the content readable. It's also a really good idea to summarize the information at that link in your answer so if the link dies the useful information is still available for future searchers. – the Tin Man May 09 '16 at 16:20
1

@MichaelC My best suggestion then would be to ask on the [RubyMotion forums](http://community.rubymotion.com/) where you'll probably get better visibility for your question. – Eric Henderson May 09 '16 at 16:32

How to prevent memory leaks in RubyMotion when reading large files in loops

2 Answers2