Parallel.For does not wait all iterations

Question

I am building an optimization program using Genetic Algorithms. I used Parallel.For in order to decrease time. But it caused a problem which is same in code below:

class Program
    {
        static void Main(string[] args)
        {
            int j=0;
            Parallel.For(0, 10000000, i =>
                {
                    j++;
                });
            Console.WriteLine(j);
            Console.ReadKey();
        }
    }

Every time i run the program above, it writes a different value of j between 0 and 10000000. I guess it doesn't wait for all iterations to finish. And it passes to next line. How am i supposed to solve this problem? Any help will be appreciated. Thanks.

Edition: Interlocked.Increment(ref j); clause solves the unexpected results, but this operation causes about 10 times much more time when i compare with normal for loop.

[Increment operator is not thread safe in C#](http://stackoverflow.com/questions/4628243/is-the-operator-thread-safe) — Andrei, Jan 27 '15 at 14:39
Is this a proof-of-concept or are you really trying to add up ten million integers? — Rawling, Jan 27 '15 at 14:57
It isn't the interlocked increment which makes the parallel loop take longer. It is the overhead of initializing/launching/tearing down the ten million threads that parallel for generates. You can't get rid of that overhead. What you can do is put real work in the loop body; when that work (e.g., a genetic algorithm one-step 'make an offspring and evaluate it') is much bigger than the thread management overhead you will win with the parallel for. With big enough pieces of work, the next trap is they may need to interlock over shared structures, and that may kill the speedup. — Ira Baxter, Jan 30 '15 at 13:01

score 5 · Accepted Answer · edited Jan 27 '15 at 15:08

5

You could use the Interlocked.Increment(int32) method which would probably be easiest.

Using Parallel.For will create multiple threads which will execute the same lambda expression; in this case all it does is j++.

j++ will be compiled to something like this j = j + 1, which is a read and write operation. This can cause unwanted behavior.

Say that j = 50.

Thread 1 is executing the read for j++ which will get 50 and will add 1 to it. Before that thread could finish the write operation to j another thread does the read operation and reads 50 from j then the first thread has finished his write operation to j making it 51 but the second thread still has 50 in memory as the value for j and will add 1 to that and again write 51 back to j.

Using the Interlocked class makes sure that every operation happens atomically.

edited Jan 27 '15 at 15:08

Servy

202,030
26
332
449

answered Jan 27 '15 at 14:41

grabthefish

962
14
30

3

you could explain why the asker's code is failing and why his guess about `Parallel.For` is incorrect which would probably be most useful – Rawling Jan 27 '15 at 14:43
Yeah it solves the incrementation, but causes much more time:( – ahmetsarias Jan 27 '15 at 14:51
@Rawling i'm writing somthing down to explain this, bit dificult to explain for me – grabthefish Jan 27 '15 at 14:57
1

@user3648161 If you care about having a fast program, just change it to `Console.WriteLine(10000000);` – Servy Jan 27 '15 at 15:01
edited with an (hopefully understandable) explanation – grabthefish Jan 27 '15 at 15:04
1

@Servy You are very clever. How couldn't I think that ! – ahmetsarias Jan 27 '15 at 15:06

score 3 · Answer 2 · answered Jan 27 '15 at 14:40

Your access to j is not syncronized. Please read a basic book or tutorial on multi-threading and syncronization.

Parallel.For does wait for all iterations.

Using syncronization (and thereby defeating the use of the parallel for):

class Program
{
    static void Main(string[] args)
    {
        object sync = new object;
        int j=0;
        Parallel.For(0, 10000000, i =>
            {
                lock(sync)
                {
                  j++;
                }
            });
        Console.WriteLine(j);
        Console.ReadKey();
    }
}

Theodoros Chatzigiannakis · Answer 3 · 2015-01-30T12:54:03.657

Parallel.For does wait for all iterations to finish. The reason you're seeing unexpected values in your variable is different - and it is expected.

Basically, Parallel.For dispatches the iterations to multiple threads (as you would expect). However, multiple threads can't share the same writeable memory without some kind of guarding mechanism - if they do, you'll have a data race and the result is logically undeterministic. This is applicable in all programming languages and it is the fundamental caveat of multithreading.

There are many kinds of guards you can put in place, depending on your use case. The fundamental way they work is through atomic operations, which are accessible to you through the Interlocked helper class. Higher-level guards include the Monitor class, the related lock language construct and classes like ReaderWriterLock (and its siblings).

Parallel.For does not wait all iterations

3 Answers3