0

I have somewhat of a complicated foreach loop. I have to throttle on every 100 connections due to the web server throwing a 529, and I know that in a parallel my throttling would mess up.

Is there a way to make this run in parallel, and still work the same? I've only just started dabbling in Parallel programming and don't think I know a way around this one.

foreach (var userId in IdCollection)
{
    i++;

    if (i % 100 == 0 && i != 0)
    {
        Thread.Sleep(15 * 1000);
    }

    ProcessId(userId, file);
}

ProcessId:

static async void ProcessId(long userId, string file)
{
    var username = await Utilities.GetUsernameFromId(userId);
    FileUtilities.WriteUsernameToFile(username, file);
    IdItemsProcessed++;
}

Method GetUsernameFromId is the one that actually hangs, my reasoning for running in parallel is to hopefully speed this up, I have over 1 million ID's to process and at this rate, it'll take me forever.

It runs a web client to a server, grabbing the response and parsing it into a username. Pretty simple stuff but when you're doing it in big numbers can become a problem.

nuzusi
  • 33
  • 3
  • You could use `Parallel` class, for example `Parallel.Invoke(() => DoSomeWork(), () => DoSomeOtherWork());` – L. Page Apr 28 '18 at 07:37
  • The parallel was a definite, the part of the question I struggled with was making parallel wait on every 100 interations as its impossible the way Parallel currently works. – nuzusi Apr 28 '18 at 07:39
  • Are you hitting rate limits on a single thread with that foreach loop and you want to try to parallelize it? or that is just some proposed logic? If you are being rate limited (429 - no idea what you mean by status code 529) it means you are calling too frequently and parallelizing is not going to help you. – Mike Zboray Apr 28 '18 at 08:10
  • It throws 529 every 100 requests. It's not because I'm calling too often. – nuzusi Apr 28 '18 at 08:12

0 Answers0