Running multiple tasks at the same time using a single source data

Question

So how can I run multiple tasks at once using a txt file as input.

Load Source Data

var lines = File.ReadAllLines("file.txt")

Run Tasks

forearch(var line in lines)
{
   //I want to execute 3 tasks and each task needs to receive a line. At the end of each task he should pick up another line that has not been used and continue to the end of the file.

}

Does it need to be a task or should it just run in parallel? Your use-case hints at the latter. In this case, use `Parallel.ForEach`. If you really need to restrict it to 3 parallel executions use the parameter `new ParallelOptions { MaxDegreeOfParallelism = 3 }`, otherwise Parallel.ForEach will dynamically adjust the number of concurrent operations. — ckuri, Apr 10 '19 at 12:19
I have async methods ... would it normally work in parallel.foreach? Do I need to tell him to pick up another line not used at the end? — Juhann, Apr 10 '19 at 12:20
You don't have to tell parallel for each to pick an unused line. It will already do that. you. Like a for loop won't iterate the same line twice. I will recommend the serie of article about pareelelisation from MSDN : https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/how-to-write-a-simple-parallel-foreach-loop — xdtTransform, Apr 10 '19 at 12:22
Possible duplicate of [How can I convert this foreach code to Parallel.ForEach?](https://stackoverflow.com/questions/12251874/how-can-i-convert-this-foreach-code-to-parallel-foreach) — xdtTransform, Apr 10 '19 at 12:24
And for [limiting the parallel](https://stackoverflow.com/questions/9290498/how-can-i-limit-parallel-foreach) to only 3 you can use the `new ParallelOptions { MaxDegreeOfParallelism = N }` — xdtTransform, Apr 10 '19 at 12:27

score 1 · Answer 1 · answered Apr 10 '19 at 12:23

1

Have you looked at Parallel.ForEach?

use like :

Parallel.ForEach(File.ReadLines("file.txt"), new ParallelOptions { MaxDegreeOfParallelism = 3 }, line => { \\ do stuff })

answered Apr 10 '19 at 12:23

Steve Todd

1,250
6
13

my parallelism is ignoring the option to create only 3 .. why? await Task.Factory.StartNew(() => { Parallel.ForEach(lines, new ParallelOptions() { MaxDegreeOfParallelism = 3 }, (current) => { CreatAccount(current); }); }); – Juhann Apr 10 '19 at 12:53
Are you sure that more than three tasks are running at any given time? the above should run only three instances at once, but may create more than three task objects to achieve this. – Steve Todd Apr 10 '19 at 13:51

score 0 · Answer 2 · answered Apr 10 '19 at 12:58

Maybe something like this:

async void Main()
{
    var lines = File.ReadAllLines("file.txt");
    int i = 0;
    var concurrency = 3;
    while (i < lines.Length)
    {
        var tasks = new List<Task>(concurrency);
        for (int j = 0; j < concurrency && i < lines.Length; j++)
        {
            tasks.Add(MyMethod(lines[i++]));
        }
        await Task.WhenAll(tasks);
    }
}
public Task MyMethod(string s)
{
    return Task.CompletedTask;
}

score 0 · Answer 3 · answered Apr 10 '19 at 20:45

you can try this:

private static async Task Main(string[] args) {
  const ushort concurrentWorkers = 5;
  var lines = File.ReadAllLines("file.txt");

  var concurrentSourceQueue = new ConcurrentQueue<string>(lines);

  var worker = Enumerable.Range(0, concurrentWorkers)
    .Select(_ => DoWorkAsync(concurrentSourceQueue));

  await Task.WhenAll(worker);
}

private static async Task DoWorkAsync(ConcurrentQueue<string> queue) {
  while (queue.TryDequeue(out var item)) {
    //process line here
  }
}

Running multiple tasks at the same time using a single source data

3 Answers3