Out of Memory Exception in Parallel.ForEach

Question

I am using Parallel.ForEach to do my job but got "Out of Memory Exception".

Parallel.ForEach(flist, (item) =>
{
    string f1 = item.Split('|')[0];
    string f2 = item.Split('|')[1];
    a = File.ReadAllText(f1);
    b = File.ReadAllText(f2); 
    Consume(a, b);
});

flist's size is 351, a and b are string, each of them has 20kb size. At a certain time, the system memory blown up.

Consume return a string list, usually around 1000 strings in each iteration.

How to deal with it?

Did you check this post: http://stackoverflow.com/questions/6977218/parallel-foreach-can-cause-a-out-of-memory-exception-if-working-with-a-enumera?rq=1 — zindarod, Apr 06 '14 at 16:14
It's very possible that you're creating 351 threads at once. Each has a stack of 1MB, plus the memory used by the file text and strings / lists etc. It's not inconceivable that it could run out of memory. Try using `ParallelOptions.MaxDegreeOfParallelism` set to 4, and see if that helps. — Baldrick, Apr 06 '14 at 16:21
--@Baldrick, I am not sure how to add ParallelOptions.MaxDegreeOfParallelism in the code. Can you modify my code? — , Apr 06 '14 at 16:23

score 3 · Accepted Answer · answered Apr 06 '14 at 16:25

3

Try replacing:

  Parallel.ForEach(flist, (item) =>
    {
        string f1 = item.Split('|')[0];
        string f2 = item.Split('|')[1];
        a = File.ReadAllText(f1);
        b = File.ReadAllText(f2); 
        Consume(a, b);
    });

With:

    Parallel.ForEach(flist, 
    new ParallelOptions { MaxDegreeOfParallelism = 4 },
    (item) =>
    {
        string f1 = item.Split('|')[0];
        string f2 = item.Split('|')[1];
        a = File.ReadAllText(f1);
        b = File.ReadAllText(f2); 
        Consume(a, b);
    });

This will prevent too many threads being created. Then you can always experiment with higher numbers and see if performance improves.

answered Apr 06 '14 at 16:25

Baldrick

11,712
2
31
35

I got another error, StackOverflowException after using the code."Process is terminated due to StackOverflowException". – Apr 06 '14 at 16:29
Do you have a recursion somewhere? – Amir Abiri Apr 06 '14 at 16:52
Yes, Consume itself is deep recursed. – Apr 06 '14 at 18:17

score 0 · Answer 2 · answered Apr 06 '14 at 16:24

0

You're reading an entire file into a single string twice per loop. That's very likely the source of your problem if the file is large.

answered Apr 06 '14 at 16:24

Martin Costello

9,672
5
60
72

No, actually I got the exception after finished several tasks. – Apr 06 '14 at 16:25

Out of Memory Exception in Parallel.ForEach

2 Answers2