Filehandle class loading whole file into memory

Question

I am trying to read data from big files sequentially. Since the files can be bigger than the available memory, I don't want the file to be completely loaded into memory.

class RecordImporter {     

    func `import`(url: URL) {
        guard let reader = RecordReader(url: url) else { return }
        self.resume(from: reader)
    }    

    private func resume(from reader: RecordReader) {
        while let _ = reader.nextData() {

        }
        //Checking memory here is about 300Mb, the test file size.
        return
    }
}

class RecordReader {   

    private var fileHandle: FileHandle

    init?(url: URL) {
        do {
            self.fileHandle = try FileHandle(forReadingFrom: url)            
        }
        catch {
            return nil
        }
    }

    func nextData() -> Data? {
        let data = self.fileHandle.readData(ofLength: 10000)
        guard !data.isEmpty else { return nil } 
        return data
    }
}

After the while loop is done, the memory is about 300Mb, even though I am reading the data in 10Kb chunks.

For some reason the chunks are not being released.

Any idea what might be happening?

You probably should read each chunk in a local *autoreleasepool.* – Possible duplicate of https://stackoverflow.com/q/40576214/1187415. — Martin R, Jun 17 '19 at 12:19

crom87 · Accepted Answer · 2019-06-17T12:44:22.657

I had tried before wrapping my use of the data in a autoreleasepool, like:


func nextData() -> Data? {
        let data = autoreleasepool  {
            let data = self.fileHandle.readData(ofLength: 10000)
            guard !data.isEmpty else { return nil } 
            return data 
         }
        return data
    } 
 }

What I didn't realize is that is the reading action the one that needs to be inside:

autoreleasepool  {
 while let data = reader.nextData() {
    print(data.count)
 }              
}

This works correctly. Thanks Martin for the hint.

Filehandle class loading whole file into memory

1 Answers1