2

I develop an application that creates a bunch of pdf reports with a sql as data srouce.

In this graphic you can see the allocated space of this application when do a job with 500,100,100 and 500 PDF Files.

allocated working set

The Problem is, that the allocated space increase after each job. So at some point the application crashes. At first the application needs 73476 KB and after the last report i need 230188 KB Memory.

This Code will be called from a Task.

public static bool Run(string id, SessionDescription session, IProgress<ProgressState> progress = null, CancellationToken cancellationToken = new CancellationToken(), Func<string,bool> prepareTargetFolder = null)
{
    SessionDescription.Validate(session);
    var pageLibrary = CreatePageLibrary(id, session.FrxFiles.Select(f => f.FullPath), session.PageDescriptions, session.Mappings);
    session.PageDescriptions = CompletePageDescriptionDataSourceNames(session.PageDescriptions, pageLibrary);

    if (cancellationToken.IsCancellationRequested) return false;

    try
    {
        var parallelOptions = new ParallelOptions() {MaxDegreeOfParallelism = 1, CancellationToken = cancellationToken};

        long count = 0;

        if (session.Jobs.Count > 0)
        {
            var job = session.Jobs.First();
            job.TargetSpec = TargetDescription.ComposeFullPath(session.Target ?? new TargetDescription(), job.TargetSpec, job.DataSourceSet, 0);
            if (prepareTargetFolder != null && !prepareTargetFolder(Path.GetDirectoryName(job.TargetSpec)))
                return false;

            ProcessOneDocument(id, session.PageDescriptions, session.PageCompositions, pageLibrary, job.DataSourceSet, session.BackpageName, job.TargetSpec);
            ProcessProgress(progress, ref count, session.Jobs.Count, job.TargetSpec);
        }

        Parallel.For(1, session.Jobs.Count, parallelOptions, index =>
        {
            var job = session.Jobs[index];
            ProcessJob(id, index, job, session, pageLibrary);
            ProcessProgress(progress, ref count, session.Jobs.Count, job.TargetSpec);
        });
    }
    catch (OperationCanceledException)
    {
        return false;
    }
    catch (AggregateException exception)
    {
        throw exception.Flatten();
    }
    finally
    {
        GC.Collect();
        GC.WaitForPendingFinalizers();
    }
    return true;
}

I added the

GC.Collect();
GC.WaitForPendingFinalizers();

but I'm not a big fan to call the GC myself. And it didn't help against the steady increase of memory usage.

I did try to analyse the used memory with ANTS Memory Profiler. The most used space it unmanaged so I don't know you is to blame.

ANTS Memory Profiler

Memory Comsumption

So any Ideas?

Update, report will be created with fastreport .net 2013.3 and i added more code.

Thx a lot! Martin

hdev
  • 6,097
  • 1
  • 45
  • 62
  • Without your code it's hard to say. In general you may not care about memory usage and GC shouldn't be explicitly called. Of course _cum grano salis_ you may need to reuse allocated memory when possible and only if you **properly disposed all disposable object** (objects that allocate unmanaged memory usually are disposable). – Adriano Repetti Nov 25 '13 at 13:56
  • Be sure to use `GC.AddMemoryPressure` if you're allocating that much unmanaged memory: http://msdn.microsoft.com/en-us/library/system.gc.addmemorypressure(v=vs.110).aspx – dcastro Nov 25 '13 at 13:56
  • Of the code you have not shown are you disposing of all your objects? – Colin Steel Nov 25 '13 at 13:56
  • If you are connecting to an external data source, or are using some kind of PDF creation API, then these areas are very likely suspects for unmanaged memory leaks! I think you might need to post more code... – Baldrick Nov 25 '13 at 14:03
  • I added the whole code. – hdev Nov 25 '13 at 14:11
  • @Baldrick Can I control the memory consumption of a referenced assembly? – hdev Nov 25 '13 at 14:37
  • The graph shows memory usage increasing with the number of PDF files processed. Therefore I'd look for code that executes repeatedly, like the `Parallel.For`, where there is a call to method `ProcessJob`: could you post this method's code? – groverboy Nov 27 '13 at 10:46

1 Answers1

1

This sounds to me like a memory leak in your application or libraries you are using. Anything that implements IDisposable should be either disposed or wrapped in a using block. Contrary to some people's belief, it is possible to have a memory leak with C#, especially if you use something that wraps native code. See this for an example of how a memory leak could occur.

Also, remember that memory usage and how the GC reclaims that varies per machine. The GC isn't guaranteed to reclaim memory until its needed, so while it may sometimes look like a memory leak is occuring (especially running on server operating systems), it actually isn't. Thats not the case here since you are getting OOM but it is something to think about that purely rising memory isn't always an indication of a leak

Community
  • 1
  • 1
devshorts
  • 8,572
  • 4
  • 50
  • 73
  • Can I unload/dispose a whole refernced assembly? In the case FastReport. So I would dump everything when the report generation has finished and I have a clean state. – hdev Nov 25 '13 at 14:34
  • No, that's overkill. Just make sure to dispose of any objects that implement IDisposable. If it has a "dispose" method on it, wrap it in a using block, or if you have to, call `.Dispose()` on it. – devshorts Nov 25 '13 at 14:38
  • So I should make all my complex types disposable which are involved in the report process? A Developer from this API suggest the use of differetn AppDomains. www. fast-report.com/en/forum/index.php?s=&showtopic=6727&view=findpost&p=18312 – hdev Nov 25 '13 at 16:21
  • The rule of thumb is that any class that instantiates an IDisposable type should also be disposable, so this way the disposable propagates through whatever uses it. If you have something that is disposable and you aren't disposing it chances are you are leaking something (memory, handles, other resources, etc). – devshorts Nov 25 '13 at 17:22
  • I disposed everything that I could. But if I try to create around 4k PDF Files I got an OOM. I changed my application to 64Bit and if this dosn't help I will take a look to the solution mentionend before, the use of new AppDomains. – hdev Nov 26 '13 at 11:45
  • Are you disposing of each pdf after you make it? If you wait till 4000 are done and then dispose of everything you may still get an OOM. – devshorts Nov 26 '13 at 20:55
  • Everything I can, will be disposed. But the 3th party framwork seems to be the problem. Now I deploy my application in 64bit so I don't get a OOM. But I really want to fix this memory consumption behavior. – hdev Nov 27 '13 at 14:26
  • If you're using a 3rd party that has a leak, there's not much you can do other than use a different library, or if its open source fix that library. – devshorts Nov 27 '13 at 15:47