How do I iterate over two lists using Foreach.Parallel

Question

I am facing quite a struggle, I want to iterate over a list using Parallel.Foreach.

So picture this

static List<string> proxyList = new List<string>();
static List<string> websiteList = new List<string>();

Each list is looking something like this

192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.4

And the website list

https://google.com
https://spotify.com
https://duckduckgo.com
https://amazon.com

I want to achieve something like this but I have no idea how, no matter how I twist and turn I cant seem to find any logic.

Foreach(var proxy in proxyList)
{
    If(proxyIsAlive)
        //Try to connect to the first website in the website list
    else
       //Try the next proxy until I get a working one 
       //and then try to connect to the most recent one
    }
 }

The issue I am facing is that I have to idea how to access the websites in the website list, I want to connect to

EDIT: this is what my logic looks like so far

private static void Connect()
{
   string tproxy = "";
   int port;

   foreach (var website in websiteList)
   {
      foreach (var proxy in proxyList)
      {
         var proxySplit = proxy.Split(':');
         tproxy = proxySplit[0];
         port = Convert.ToInt32(proxySplit[1]);
         //if(ProxyIsAlive)

      }
      //Use that proxy down here to connect
   }
}

I only want to move out of the proxy foreach IF ProxyIsAlive returns true

@Saruman It's not that its not working its that I cant find the logic to how to structure the thing I want to structure — Aleks Slade, Dec 27 '17 at 02:35
What do you want to do? check if a proxy works and do some other actions all in parallel if it does? — TheGeneral, Dec 27 '17 at 02:37
Imagine this, I have a list of proxies, I want to check 4 at a time if they are working, hence the parallel with the option of 4, and then if it finds one that works, I want to connect to a website using that working one, so let's say 2 out of those 4 worked, I want to connect to two websites using those proxies, the websites are all different and they are stored in a list. Am I making any sense? — Aleks Slade, Dec 27 '17 at 02:39
The first unused website in the list? (ie block access to that website until it is finished) or access them sequentially until they have all been used once — TheGeneral, Dec 27 '17 at 03:16
I want to access each website in the website list with a unique proxy from the proxy list, in my head it should get the proxies with a foreach because that seems to be logical, — Aleks Slade, Dec 27 '17 at 03:21
So in essence, you want to go to each website and use a working proxy and if so check the most recent working proxy first?\ — TheGeneral, Dec 27 '17 at 03:25
Do you want to check to see if the proxy is alive each time, is that IO call i.e call to the internet? — TheGeneral, Dec 27 '17 at 03:41

TheGeneral · Accepted Answer · 2017-12-27T04:34:49.347

Notes :

This uses nested Parallel as per your original question
It assumes you want to check the last good proxy first
If a good proxy is found it updates the success time
It processes each website and proxy in parallel

Note : Parallel.ForEach is suited for CPU-bound tasks, you need to be careful you aren't just wasting resources blocking threads waiting for IO operations to complete

Class to hold proxy info

public class Proxy
{
   public string Host { get; set; }
   public int Port { get; set; }
   public DateTime LastSuccess { get; set; }

   public Proxy(string value)
   {
      var proxySplit = value.Split(':');
      Host = proxySplit[0];
      Port = Convert.ToInt32(proxySplit[1]);
      LastSuccess = DateTime.MinValue;
   }
}

Code to run in parallel

var proxies = proxyList.Select(x => new Proxy(x)).ToList();

Parallel.ForEach(webSites, new ParallelOptions { MaxDegreeOfParallelism = 4 }, site =>
   {
      Parallel.ForEach(proxies.OrderByDescending(x => x.LastSuccess), new ParallelOptions { MaxDegreeOfParallelism = 4 }, proxy =>
      {               
         if(!CheckProxy(proxy))
         {
            //check next proxy
            return;
         }
         // if we found a good proxy
         // update the lastSuccess so we check that first
         proxy.LastSuccess = DateTime.Now;
         // do something to the website 
      });
   });
}

Note : This may not be the best approach, if you have CPU-bound code, parallelism is appropriate; if you have I/O-bound code, asynchrony is appropriate. In this case, an HttpClientExtension.GetHttpResponse is clearly I/O, so the ideal consuming code would be asynchronous.

I would consider looking up the topics Parallel execution for IO bound operations

Existing SO questions deal with this Such as

Parallel execution for IO bound operations

Parallel.ForEach vs Async Forloop in Heavy I/O Ops

score 0 · Answer 2 · answered Dec 27 '17 at 03:55

So this is what I ended up doing.

private static void ConnectToWebsite()
        {
            var proxyIP = "";
            int port;

            foreach (var website in WebsiteList)
            {
                foreach (var proxy in proxyList)
                {
                    var proxySplit = proxy.Split(':');
                    proxyIP = proxySplit[0];
                    var convert = Int32.TryParse(proxySplit[1], out port);
                    if(HttpClientExtension.GetHttpResponse(getCMYIP, proxyIP, port))
                        Console.WriteLine(website + proxy);
                }
            }
        }

That will check proxies until it finds a working one. Now I need to make this async to speed things up.

How do I iterate over two lists using Foreach.Parallel

2 Answers2