Why is my Parallel.Foreach appear not to be using 10 threads?

Question

There does not seem to be a MinDegreeOfParallelism. The following code only seems to use 1% cpu, so I suspect it is NOT using cores properly:

Parallel.ForEach(lls, new ParallelOptions { MaxDegreeOfParallelism = 10 },  GetFileSizeFSO);

Is there a way to FORCE using 10 cores/Threads?

Additional information:

private void GetFileSizeFSO(List<string> l)
{
    foreach (var dir in l)
    {

        var ds = GetDirectorySize3(dir);
        Interlocked.Add(ref _size, ds);
    }
}

    public static long GetDirectorySize3(string parentDirectory)
    {
        Scripting.FileSystemObject fso = new Scripting.FileSystemObject();
        Scripting.Folder folder = fso.GetFolder(parentDirectory);
        Int64 dirSize = (Int64)folder.Size;

        Marshal.ReleaseComObject(fso);


        return dirSize;
    }

Possible duplicate of [Parallel.For not utilising all cores](http://stackoverflow.com/questions/32188185/parallel-for-not-utilising-all-cores) — , Mar 04 '16 at 09:20
`GetFileSizeFSO` sounds like something IO related. Does this method do any CPU work? — Yuval Itzchakov, Mar 04 '16 at 09:20
Why is it important for you to use all 10 cores? Do you have performance problems? Usually it's not the best idea to tell thread pool how to work. It does it's job not bad without your "help". — Sasha, Mar 04 '16 at 09:21

score 2 · Answer 1 · answered Mar 04 '16 at 09:25

2

What does your function GetFileSizeFSO do? In case it accesses files on disc, that must be your main time-consumer. Processor is simply too fast and disc can't catch up with the processor. So processor have enough time to spare and wait while HDD completes it's job.

If you need to optimize your code, you better look into accessing files more efficiently than trying to load processor for 100%.

answered Mar 04 '16 at 09:25

Sasha

8,537
4
49
76

I have added the the additional code to my question. I have very large folders with huge numbers of file, and subl-folders. I am trying to work out the Size of folder FAST. This is very, very slow - so I am using FSO and now trying to make it parallel too. – ManInMoon Mar 04 '16 at 09:55
@ManInMoon, have you tried doing this non-parallel (consequtively)? It might appear to be significantly faster than parallel solution because HDD can't read data in parallel - it will jump from one place to another every time different thread asks something and this may result in the same situation as if you copy 2 folders as 2 different parallel jobs - it's many times slower than copying them one by one. – Sasha Mar 04 '16 at 09:59
Also you might want to check related threads: http://stackoverflow.com/questions/9831641/more-efficient-method-of-getting-directory-size http://stackoverflow.com/questions/468119/whats-the-best-way-to-calculate-the-size-of-a-directory-in-net http://stackoverflow.com/questions/2869561/what-is-the-fastest-way-to-calculate-a-windows-folders-size – Sasha Mar 04 '16 at 10:03
Actually, I have already seen those threads. I have done a lot of testing and FSO and parallel is the best combination I have found so far. I am just trying to balance the load to avoid problems as you suggest. – ManInMoon Mar 04 '16 at 10:07

score 2 · Answer 2 · answered Mar 04 '16 at 09:25

It's called MaxDegreeOfParallelism, not MinDegreeOfParallelism. Parallel is designed for CPU-bound work - there's no point whatsoever in using more threads than you have CPUs. It sounds like your work is I/O bound, rather than CPU-bound, so Parallel simply isn't the right tool for the job.

Ideally, find an asynchronous API to do what you're trying to do - this is the best way to use the resources you have. If there's no asynchronous API, you'll have to spawn those threads yourself - don't expect to see CPU usage, though. And most importantly, measure - it's very much possible that parallelizing the workload doesn't improve throughput at all (for example, the I/O might already be saturated).

score 0 · Answer 3 · answered Mar 04 '16 at 09:26

The simple answer is - you can't.

But why should you? .NET is pretty good at choosing the optimal amount of threads used. The use of MaxDegreeOfParallelism is to limit parallelism, not to force it, for example if you don't want to give all system resources to the loop.

As a side note, judging from your function name GetFileSizeFSO, I would guess that it reads file sizes from your persistent storage, which would explain why your CPU is not being fully used.

score 0 · Answer 4 · answered Apr 12 '17 at 16:30

ManInMoon, your CPU usage is probably slow because the meat of the work you're doing is probably bound by your storage mechanism. 10 cores hitting the same hard drive for getting file sizes may not be any faster than 2 cores, because going and making a hit against a hard drive is a relatively (ridiculously) more expensive operation than the surrounding C# logic you have there.

So, you don't have a parallelism problem, you have an I/O problem.

Side note, perhaps don't use FSO, use .NET's FileInfo instead.

Why is my Parallel.Foreach appear not to be using 10 threads?

4 Answers4