Multi Thread Processing with Parallel.Foreach c# .net4.5.1

Question

I have an IEnumerable<customClass> object that has roughly 10-15 entries, so not a lot, but I'm running into a System.IO.FileNotFoundException when I try and do

Parallel.Foreach(..some linq query.., object => { ...stuff....});

with the enumerable. Here is the code I have that sometimes works, other times doesn't:

        IEnumerable<UserIdentifier> userIds = script.Entries.Select(x => x.UserIdentifier).Distinct();

        await Task.Factory.StartNew(() =>
        {
            Parallel.ForEach(userIds, async userId =>
            {
                Stopwatch watch = new Stopwatch();
                watch.Start();

                _Log.InfoFormat("user identifier: {0}", userId);
                await Task.Factory.StartNew(() =>
                {
                    foreach (ScriptEntry se in script.Entries.Where(x => x.UserIdentifier.Equals(userId)))
                    {
                        //    // Run the script //
                        _Log.InfoFormat("waiting {0}", se.Delay);
                        Task.Delay(se.Delay);
                        _Log.InfoFormat("running SelectionInformation{0}", se.SelectionInformation);

                        ExecuteSingleEntry(se);
                        _Log.InfoFormat("[====== SelectionInformation {1} ELAPSED TIME: {0} ======]", watch.Elapsed,
                            se.SelectionInformation.Verb);
                    }

                });

                watch.Stop();
                _Log.InfoFormat("[====== TOTAL ELAPSED TIME: {0} ======]", watch.Elapsed);
            });
        });

When the function ExecuteSingleEntry is ran, there is a function a few calls deep within that function that creates a temp directory and files. It seems to me, that when I run the parallel.foreach the function is getting slammed at once by numerous calls (I'm testing 5 at once currently but need to handle about 10) and isn't creating some of the files I need. But if I hit a break point in the file creation function and just F5 every time it gets hit I don't have any problems with a file not found exception being thrown.

So, my question is, how can I achieve running a subset of my scripts.Entries in parallel based on the user id within the script entries with a delay of 1 second between each different user id entries being started?

and a script entry is like:

UserIdentifier: 141, SelectionInformation: class of stuff, Ids: list of EntryIds, Names: list of Entry Names

And each user identifier can appear 1 or more times in the array. I want to start all the different user identifiers, more or less, at once. Then Task out the different SelectionInformation's tied to a script entry.

scripts.Entries is an array of ScriptEntry, which is as follows:

    [DataMember]
    public TimeSpan Delay { get; set; }

    [DataMember]
    public SelectionInformation Selection { get; set; }

    [DataMember]
    public long[] Ids { get; set; }

    [DataMember]
    public string Names { get; set; }

    [DataMember]
    public long UserIdentifier { get; set; }

I referenced: Parallel.ForEach vs Task.Factory.StartNew to obtain the

Task.Factory.StartNew(() => Parallel.Foreach({ }) ) so my UI doesn't lock up on me

why don't you try making `ExecuteSingleEntry(se);` thread safe — Sam I am says Reinstate Monica, Jul 02 '15 at 16:52
If all you do in the `Parallel.ForEach` loop is creating some files and directories then you can do it sequentially just as well. The speed may be even higher then. The mass storage performance not the CPU is your bottleneck. — Dzienny, Jul 02 '15 at 17:00
Depending on what `ExecuteSingleEntry` actually does, you can try putting the file creation and deletion logic in a [lock statement](https://msdn.microsoft.com/en-us/library/c5kehkcz.aspx) — Sam I am says Reinstate Monica, Jul 02 '15 at 17:01
The `ExecuteSingleEntry` does a lot more than create files and directories. It goes out to an oracle database and runs reports against the database. `.wsc` files are created and dumped into directories to analysis on the returned data. I'm attempting to lock the file creation at the moment. — B-M, Jul 02 '15 at 17:06
The parallel.foreach is intended to reproduce 3-5 users per machine running analysis, at once, on a (semi)large collection of data and the execute single entry function chooses the analysis path based on the selection information. — B-M, Jul 02 '15 at 17:08
@B-M if you're trying to simulate multiple users at once, then you need to focus on making `ExecuteSingleEntry(se);` thread safe — Sam I am says Reinstate Monica, Jul 02 '15 at 17:42

score 1 · Accepted Answer · answered Jul 02 '15 at 19:48

There are a few principles to apply:

Prefer Task.Run over Task.Factory.StartNew. I describe on my blog why StartNew is dangerous; Run is a much safer, more modern alternative.
Don't pass an async lambda to Parallel.ForEach. It doesn't make sense, and it won't work right.
Task.Delay doesn't do anything by itself. You either have to await it or use the synchronous version (Thread.Sleep).

(In fact, in your case, the internal StartNew is meaningless; it's already parallel, and the code - running on a thread pool thread - is trying to start a new operation on a thread pool thread and immediately asynchronously await it???)

After applying these principles:

await Task.Run(() =>
{
  Parallel.ForEach(userIds, userId =>
  {
    Stopwatch watch = new Stopwatch();
    watch.Start();

    _Log.InfoFormat("user identifier: {0}", userId);
    foreach (ScriptEntry se in script.Entries.Where(x => x.UserIdentifier.Equals(userId)))
    {
      //    // Run the script //
      _Log.InfoFormat("waiting {0}", se.Delay);
      Thread.Sleep(se.Delay);
      _Log.InfoFormat("running SelectionInformation{0}", se.SelectionInformation);

      ExecuteSingleEntry(se);
      _Log.InfoFormat("[====== SelectionInformation {1} ELAPSED TIME: {0} ======]", watch.Elapsed,
          se.SelectionInformation.Verb);
    }

    watch.Stop();
    _Log.InfoFormat("[====== TOTAL ELAPSED TIME: {0} ======]", watch.Elapsed);
  });
});

Multi Thread Processing with Parallel.Foreach c# .net4.5.1

1 Answers1