1

I have a web service that i am using to submit a text message to be sent as a bulk sms. I have been given a url that i should submit the text message to for sending.

I have the numbers in a csv file that i am reading in this way

$row = 1;
if (($handle = fopen("27k.csv", "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
        $num = count($data);
        $row++;
        for ($c=0; $c < $num; $c++) {
           echo $data[$c] . '<br/>';
        }
    }
    fclose($handle);
}

The messages are being submitted one by one to web service and sending 30,000 records is taking hours.

Is there a programming construct in any language that i can use to be able to make 30,000 concurrent requests to the web service instead of sending one text at a time?.

  • Where are you sending here requests here? Asynchronous calls are common, so be more specific - do you want to use PHP for this? [here](http://stackoverflow.com/questions/124462/asynchronous-php-calls#2924987) is an example for php posting asynchronously. – kabanus May 02 '17 at 12:43
  • @kabanus as the comments in that post point out, that is not a way to perform an asynchronous request – dm03514 May 02 '17 at 12:50
  • @dm03514 Correct! Sorry - I didn't actually read it, I just assumed all the up-votes mean it's OK. Thanks. – kabanus May 02 '17 at 12:51

1 Answers1

0

Yes! There are many concurrency constructs that could help you:

Each of the above has its own tradeoffs. With threading memory synchronization has to be considered (often simplified by passing data between threads with queues).

Event loops are optimized for IO (should be perfect for your use case) but representing futures and wrapping head around asynchronous operations, yielding, and what state the event loop is in in any given time can be maddening.

Interprocess job queue has operational overhead of managing a queue and a pool of workers.

While any of the above solutions should allow for X number of concurrent requests, it's generally good practice to bound your resource pools (ie limit concurrency). If you launch 30,000 concurrent requests (can your machine even launch that many???) You could easily overwhelm a server. If the launch is driven by an end user or an uncontrolled action, you could easily have X * 30000 requests being launched at any time.

Also what happens if a single request fails? will you have to retry? What about when you get rate limited?

dm03514
  • 54,664
  • 18
  • 108
  • 145
  • I have considered threads and a job queue. For threads, since i will be reading from a csv file, i several threads reading the same file may result in duplicate texts. and so is a job queue since there will be multiple workers. –  May 02 '17 at 12:49
  • @GeoffreyIsle one way to address this, is to have a single isolated thread/process read the file and submits each url in the queue (either intra process queue or interprocess queue) and have a pool of workers (threads, processes) consuming off the queue, working concurrently. The task that reads the file and submits to the queue is called the producer, and then it fans out the requests (round robin/random, guaranteed by pretty much all queues) to the workers (consumers). Which will NOT result in duplicates – dm03514 May 02 '17 at 12:51
  • That arrangement sounds good. I was thinking something along those lines too but the idea of a queue does not fill me up with enthusiasm. but i shall try it out with beanstalk. I am looking at http://elixir-lang.org/ elixir and wondering if it can handle the 30,000 requests without using so much resources. –  May 02 '17 at 12:58