TFileStream read huge files piece by piece

Question

Earlier today I opened a question here asking if my method to scan files in computer was correct. As solution, I received a few tips, and the one of the solutions I thought: "this need to be solved urgent!", was saying about memory overflow, once I was reading the files entirely in memory. So I started trying to find a way to read the files piece by piece, and I got something (wrong/bogus), that I need some help to figure out how to do this correctly. The method is simple like this for now:

procedure ScanFile(FileName: string);
const
  MAX_SIZE = 100*1024*1024;
var
  i, aux, ReadLimit: integer;
  MyFile: TFileStream;
  Target: AnsiString;
  PlainText: String;
  Buff: array of byte;
  TotalSize: Int64;
begin
  if (POS('.exe', FileName) = 0) and (POS('.dll', FileName) = 0) and
      (POS('.sys', FileName) = 0) then //yeah I know it's not the best way...
    begin
      try
        MyFile:= TFileStream.Create(FileName, fmOpenRead);
      except on E: EFOpenError do
        MyFile:= NIL;
      end;
      if MyFile <> NIL then
      try
        TotalSize:= MyFile.Size;
        while TotalSize > 0 do begin
          ReadLimit:= Min(TotalSize, MAX_SIZE);
          SetLength(Buff, ReadLimit);
          MyFile.ReadBuffer(Buff[0], ReadLimit);
          PlainText:= RemoveNulls(Buff); //this is to transform the array of bytes in string, I posted the code below too...
          for i:= 1 to Length(PlainText) do
            begin //Begin the search..
            end;
          dec(TotalSize, ReadLimit);
         end;
  finally
    MyFile.Free;
  end;
end;

Code for RemoveNulls is:

function RemoveNulls(const Buff: array of byte): String;
var
  i: integer;
begin
  for i:= 0 to Length(Buff) do
    begin
      if Buff[i] <> 0 then
        Result:= Result + Chr(Ord(Buff[i]));
    end;
end;

Ok, the problems I got with this code so far was:

1- each time the while is repeated, I get more memory consumed, when I was expecting to get only MAX 100MB as described in the MAX_SIZE variable, right?

2- I created a file with 2 occurrences of what should be filtered, and for some unknown reason I got about 10 repeated occurrences, looks like I'm scanning the file repeatedly.

I appreciate your help guys, and if someone have this kind of code already done, post here please, I don't pretend to re-create the wheel...

Answer 1

I'd say that RemoveNulls is your problem. Suppose that you just read 100MB into a string that you passed to RemoveNulls. You would then allocate a string of length 1. The reallocate to length 2. Then to length 3. Then to length 4. And so on, all the way to length 100*1024*1024.

That process will fragment your memory, as well as being appallingly slow. Heap allocation is to be avoided when performance matters. You've no need for it at all. Read a chunk of the file, and search directly in the buffer that you read.

There are various problems with your code that I can see:

1. Your file extension check is broken, as I described in your previous question.
2. You are not handling exceptions correctly, as I described in your previous question.
3. Your for loop in RemoveNulls has buffer overrun. Loop from low() to high().

It's not possible to comment on the search code since that's not present in the question.