How to break big foreach loop into batch and process it simultaneously?

Question

foreach (var deviceId in deviceList)
{
    // register device into IoT hub 
    Device device;
    RegistryManager registryManager = RegistryManager.CreateFromConnectionString("connectionString");
    device = await registryManager.AddDeviceAsync(new Device(deviceId));

    // send message to iot hub
    DeviceClient deviceClient;
    await deviceClient.SendEventAsync("data");                       
}

if device are 10000 then how can i break it into multiple batch and process it?

I tried this code but its not promissing

public IEnumerable<user> GetBatch(int pageNumber)
{
    return users.Skip(pageNumber * 1000).Take(1000);
}

If you want to process "simultaneously" (in parallel), you could look at parellelizing the foreach. See: https://stackoverflow.com/questions/11564506/nesting-await-in-parallel-foreach for some approaches with `Async` and `Await`. If you want to chunk it into batches, `.Skip` and `.Take` are indeed the right way to do that. — Jonathan, Dec 12 '18 at 17:11
Please, post working code. In this example `deviceClient == null`, so will be NRE. — Alexander Petrov, Dec 13 '18 at 02:07
`AddDeviceAsync` and `SendEventAsync` are CPU-bound or IO-bound operations? In the first case, it will be good to work in parallel, in the second case, need asynchronous code. — Alexander Petrov, Dec 13 '18 at 02:10
Is the `RegistryManager` thread safe? Can we add devices from different threads at the same time, or do we need lock? — Alexander Petrov, Dec 13 '18 at 02:24

Steve Danner · Accepted Answer · 2018-12-12T17:21:26.677

7

This is a great case for a Parallel.ForEach Loop, which will automatically distribute your loop processing across multiple threads. Very easy to re-arrange your code into such a loop and utilize the built-in Parallel library to enable parallel processing. This assumes of course that sequence doesn't really matter (and it doesn't seem to based on what little we can see).

EDIT: If you do need specific batches of 100 or 200 as noted in your comment, you can use the System.Collections.Concurrent.Partitioner class to break a parallel loop up as desired, this SO post actually does a good job describing how to use it.

Parallel.ForEach(devices, (device) =>
            {
                        // register device into IoT hub 
                        Device device;
            RegistryManager registryManager = RegistryManager.CreateFromConnectionString("connectionString");
                        device = await registryManager.AddDeviceAsync(new Device(deviceId));


                        // send message to iot hub
                         DeviceClient deviceClient;
                        await deviceClient.SendEventAsync("data");                       

            });

edited Dec 12 '18 at 17:21

answered Dec 12 '18 at 17:10

Steve Danner

21,818
7
41
51

but here we have not process it into batch . batch of 100 or 200. before Parallel.ForEach it will be great if we can batch it. can you help me? thanks – Neo Dec 12 '18 at 17:15
1

Correct, it's not broken into specific batches. Can you share some more info in your post about the exact desired end result? Is there a reason you want it into specific batches of 100 or 200? – Steve Danner Dec 12 '18 at 17:17
i want to into batches because it is taking to much time to execute more no of device list. i do not want it into 100 or 200 any no is good for me. – Neo Dec 12 '18 at 17:19
1

In that case, the parallel loop as originally posted will do what you want. It will use built-in .NET algorithms to break the processing up as efficiently as possible based on threads available on the current machine. – Steve Danner Dec 12 '18 at 17:22
1

Neo, i believe you do not understand the concept of parallel.foreach. This is braking things up so it is handled in parallel and is then much faster. Yoy wanted it faster so i believe Steve helped you – Aldert Dec 12 '18 at 17:31
I highly doubt this is effective code. Tasks will be spent most of the time in awaiting. – Alexander Petrov Dec 13 '18 at 02:14
In addition, this code may be non-thread safe. Here we need clarification the author of the question. – Alexander Petrov Dec 13 '18 at 02:27

How to break big foreach loop into batch and process it simultaneously?

1 Answers1