I wrote a simple script with 200 concurrent threads that makes http requests and processes the results. I need to do around 50,000,000 requests. Script is simple modification of "twistedless solution" suggested here (the first reply).
If I want to run it on a 10-nodes AWS cluster, how can I parallelize it? If I just copy that script onto one node and run it, what happens? Same as on a local machine?
Being new to all this, I am need a starting point. Thanks!
On AWS you can have an Elastic Load Balancer (ELB). You can set up everything manually or within an AutoScaling Group. Either way, your X ( X=10 :D ) Instances will be attached to the ELB. You need to set up the ELB Health check properly, so it sees the Instances healthy and serves traffic for all of them. If this is done, you need to send your concurrent requests to the ELB endpoint. Also worth to check Sticky sessions if you need it or it causes false result, like sending all the traffic to one Instance because the source of all the requests are the same.
