Control number of threads (not Task)

Question

i have a list of files, every file that i need to run (PCAP file - transmit the packets) has its own running time.

because i want the option to handle more than 1 file parallel i am using this function that get IEnumerable<string> source and MAX number of parallel threads:

  public void doWork(IEnumerable<string> _source, int parallelThreads)
    {
        _tokenSource = new CancellationTokenSource();
        var token = _tokenSource.Token;
        Task.Factory.StartNew(() =>
        {
            try
            {
                Parallel.ForEach(_source,
                    new ParallelOptions
                    {
                        MaxDegreeOfParallelism = parallelThreads //limit number of parallel threads 
                    },
                    file =>
                    {
                        if (token.IsCancellationRequested)
                            return;
                        //process my file...
                    });
            }
            catch (Exception)
            { }

        }, _tokenSource.Token).ContinueWith(
                t =>
                {
                    //finish all the list...
                }
            , TaskScheduler.FromCurrentSynchronizationContext() //to ContinueWith (update UI) from UI thread
            );
    }

for example if i have list of 10 files and my max number of parallel threads is 4 so my program start to transmit 4 files parallel and after the first file finish another 1 file automatic start and this is works fine if i transmit all my list 1 time.

after added the option to play all the list in loop i have a problem, if i want to play for example all the list twice, after the first loop end the second begin and in this loop after the first file finish all the UI stuck and not respond. i have talk with friend of mind that he is C# developer and he told me that is probably Task known issue that sometimes get into deadlock. is it possible to use another Class instead of Task ?

1. Do something with the Exceptions, it might give you a good hint what's going wrong. 2. When the UI freezes, it's a problem on the main thread. Therefore please add your code that you left out as I guess the problem will be there somewhere. — Akku, Aug 01 '13 at 08:59
Never heard of this one *"Task known issue that sometimes get into deadlock"*, but I have seen people using (waiting for) shared resource from different tasks that resulted in a deadlock. — oleksii, Aug 01 '13 at 09:02

NeddySpaghetti · Answer 1 · 2013-08-01T23:40:43.243

0

You shouldn't be using Parallel.ForEach for file IO. It is not a cpu intensive task. You should be able to just start all the tasks one after the other. This way you will use much less threads and your application will be more scalable.

updated example:

public static void doWork(IEnumerable<string> _source, int numThreads)
{
    var _tokenSource = new CancellationTokenSource();

    List<Task> tasksToProcess = new List<Task>();
    foreach (var file in _source)
    {
        tasksToProcess.Add( Task.Factory.StartNew(() =>
                              {
                                  Console.WriteLine("Processing " + file );
                                  //do file operation
                                  Thread.Sleep(5000);

                                  Console.WriteLine("Finished Processing " + file);
                              },
                          _tokenSource.Token));

        if(tasksToProcess.Count % numThreads == 0)
        {
            Console.WriteLine("Waiting for tasks");

            Task.WaitAll(tasksToProcess.ToArray(), _tokenSource.Token);

            Console.WriteLine("All tasks finished");   
            tasksToProcess.Clear();
        }
    }             
}

void Main()
{
    var fileList = Enumerable.Range(0, 100).Select (e => "file" + e.ToString());
    doWork(fileList, 4);

    Console.ReadLine(); 
}

edited Aug 01 '13 at 23:40

answered Aug 01 '13 at 10:18

NeddySpaghetti

13,187
5
32
61

Can i have an example how to do it ? (include how to control the number of concurrent treads) – user2214609 Aug 01 '13 at 10:38
I've added a simple example to my answer. Replace `Thread.Sleep` with whatever file processing you need to do – NeddySpaghetti Aug 01 '13 at 11:12
@user2214609 As Ned said you are doing I/O work, not CPU-bound computations. So your threads are all going to sit idle most of the time anyway. The .net thread pool is quite smart about this and will manage the number of threads and the degree of concurrency for you automatically. So why do you want to control precisely the number of threads? – jods Aug 01 '13 at 11:23
@user2214609 To add a clarification to my previous comment: a task is not equivalent to a thread. In this case there will be much less threads than tasks (i.e. files). So you should be talking of degree of parallelism rather than threads, if you want to control something. – jods Aug 01 '13 at 11:24
But in your example i cannot control the concurrent number of files – user2214609 Aug 01 '13 at 12:53
I updated my example to do that. Might be better done with some kind of LINQ chunking operator like this http://stackoverflow.com/questions/419019/split-list-into-sublists-with-linq – NeddySpaghetti Aug 01 '13 at 23:42

Control number of threads (not Task)

1 Answers1