How to prevent php curl from multi connecting

Question

I built an php cli script that ouputs tweets from twitter streaming api to the commandline with cURL. When I start the script it begins outputting tweets, but at every 50th tweet I get the following error: "Easy there, Turbo. Too many requests recently. Enhance your calm.", no headers, no other ouput, just a string.

I assume I make too many connections, but I only init the cURL once. Here's my code:

//initialize cURL connection
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://stream.twitter.com/1.1/statuses/filter.json?  follow=34572451,27260086');
curl_setopt($ch, CURLOPT_USERPWD, 'USERNAME:PASSWORD');
curl_setopt($ch, CURLOPT_WRITEFUNCTION, 'saveTweet');
curl_setopt($ch, CURLOPT_MAXCONNECTS, 1);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_FORBID_REUSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Cache-Control: max-age=0', 'Connection: keep-alive', 'Keep-Alive: 3600'));

//then loop the execution to retrieve tweets
while(1){
    echo $i++.'->'; //echo number, so we know it stops at 50

    try {
        //calls saveTweet() as callback, which only purpose is to output the tweet
        curl_exec($ch); 
    }
    catch(Exception $e) {
        //echo error described above
        echo $e->getMessage(); 
        break;
    }
}

How do I prevent cURL from making too many requests (if it does that, I don't really know the exact problem)? Or how do I prevent getting the above error?

What do you mean by TPS? I tried putting a sleep (1 to 30 seconds) in the whileloop, but that didn't work. — Jelle van der Coelen, Oct 18 '12 at 12:48

score 1 · Accepted Answer · answered Oct 18 '12 at 12:48

1

You can run endless loop like that you need to sleep and usleep once in a while

sleep(1);

You would also like to look at count option count :

Specifies the number of records to retrieve. Must be less than or equal to 200. Defaults to 20.

This way you can pull more record with a single request ...

answered Oct 18 '12 at 12:48

Baba

94,024
28
166
217

I tried the sleep(). Didn't work. I'm not trying to do multiple requests, I'm trying to do the opposite - keeping a streaming connection open (Twitter Streaming API, not the REST API) – Jelle van der Coelen Oct 18 '12 at 12:59
If its a `Streaming API` then you don't need to loop : Use this :http://stackoverflow.com/a/12915348/1226894 – Baba Oct 18 '12 at 13:08
Wow thanks. That's working. I stopped looping and return the strlen() in the callback function. Now it keeps running :-) – Jelle van der Coelen Oct 18 '12 at 13:18

score 0 · Answer 2 · answered Oct 18 '12 at 13:00

0

Apparently you're opening too many connections:

curl_setopt($ch, CURLOPT_FORBID_REUSE, true);

Can you pool the connections for reuse and test it?

TPS: Transactions/Second DOS: Denial of Service; with using up resources

answered Oct 18 '12 at 13:00

जलजनक

3,072
2
24
30

I did try to set CURLOPT_FORBID_REUSE to false (even removed it from the options), but I got the same thing. – Jelle van der Coelen Oct 18 '12 at 13:06

How to prevent php curl from multi connecting

2 Answers2