OmniThreadLibrary memory leak (consumption) on pipeline running from another thread

Question

I'm running pipeline (thread's pipeline from OmniThreadLibrary) from another thread and got memory leak or rather memory consumption. But when application close then it's ok and there are no memory leak report (ReportMemoryLeaksOnShutdown := True;).

Here example: click button 10 times and test app will get ~600 MB of memory. Windows 7 x64, Delphi XE6, latest omni source.

It's a bug? Or I need use another code?

uses
  OtlParallel,
  OtlCommon;

procedure TForm75.Button1Click(Sender: TObject);
begin
  // run empty pipeline from another threads
  Parallel.&For(1, 100).Execute(
    procedure(value: integer)
    var
      pipe: IOmniPipeline;
    begin
      pipe := Parallel.Pipeline
        .Stage(procedure(const input: TOmniValue; var output: TOmniValue) begin end)
        .Run;
      pipe.Cancel;
      pipe.WaitFor(100000);
      pipe := nil;
    end
  );
end;

Edit 1: Tested that code with ProcessExplorer and found what threads count at runtime is constant, but handles count is grown. If I'm insert Application.ProcessMessages; at the end of "for loop" (after pipe's code) then test app running good, handles are closing and memory consumption is constant. Don't know why.

Answer 1

How many threads does it create ?

Check it in SysInternals Process Explorer for example. Or in Delphi IDE (View -> Debug Windows -> Threads)

I think that because you block each For-worker for wuite a long WaitFor your application then creates many worker threads for every button click, and when you click it 10 times it consequently creates 10 times many threads.

And yes, in general-purpose operating systems like Windows threads are expensive! Google for "windows thread memory footprint" - and multiply it by the number of threads created by 10 parallel-for loop you spawn.

• https://msdn.microsoft.com/en-us/library/windows/desktop/ms686774.aspx
• https://blogs.technet.microsoft.com/markrussinovich/2009/07/05/pushing-the-limits-of-windows-processes-and-threads/

This fact was the reason that for making highly parallel server applications special approaches were done to create light-eight application-level threads and bypass OS threads, to name a few

• Make special language that would spawn dozens of thousands of cooperative thread and cross-thread enforce memory safety by strict language rules: https://www.erlang.org/docs
• Make a library, which cannot enforce those regulations but at least can demand programmer to follow them voluntarily: https://en.wikipedia.org/wiki/Actor_model
• Fibers: no-protection threads within threads: What is the difference between a thread and a fiber?

However OTL being a generic library for generic threads imposes little restrictions but relies on OS-provided native threads, and they are heavy expensive in both CPU time needed to create/release Windows Threads (mitigated by Thread Pools concept) and by memory footprint needed to maintain each Windows Threads by OS (which is unavoidable and you see its manifestation).

Of course later, when all those loops are worked through, their threads are getting closed and released, together with the memory that was used to maintain them. So no memory leak indeed, once you wait enough for all your threads to be terminated - they are, with all the temporarily allocated memory they used as their workplaces.

UPD. How to check that hypothesis? easiest way would be to change how many threads is spawned by every instance of For-Loop (by every button click).

See the .NumTasks options of you Parallel.For object:

http://otl.17slon.com/book/chap04.html#leanpub-auto-iomniparallelsimpleloop-interface

By default every button click should spawn one thread for every CPU core. But you can enforce your own size of thread pool. Add .NumTasks(1) call and check memory consumption, then check it into .NumTasks(10) and do it again. If the memory consumption would grow approximately tenfold after that - then it is it.