Libcurl Callback Function Might Not be Working for Twitter Streaming

Question

Possible Duplicate:
Using HTTP authentication with libcurl in C for Twitter Streaming

I was able to write a C code that receives a stream of Tweets from streaming API. But the stream is not being put out (no output) by the code below. The code works when CURLOPT_URL is set to google.com or 9gag.com. I guess the problem has something to do with the steady stream of tweets with tremendous received data. The write_func callback function, which purpose is to print the response (stream), might not be working that is why there is no output? I am thinking that the callback function probably is being overwhelmed by the tremendous stream sent by Twitter API. Then if this is the case, how should I write the proper write callback function?

You might ask. I verified that the reception of stream is working because I watch my System Monitor's network history rise on received bytes whenever I execute the code.

Thanks!

The code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <curl/curl.h>

struct string {
  char *ptr;
  size_t len;
};

void init_string(struct string *s) {
  s->len = 0;
  s->ptr = malloc(s->len+1);
  if (s->ptr == NULL) {
    fprintf(stderr, "malloc() failed\n");
    exit(EXIT_FAILURE);
  }
  s->ptr[0] = '\0';
}

size_t writefunc(void *ptr, size_t size, size_t nmemb, struct string *s)
{
  size_t new_len = s->len + size*nmemb;

  s->ptr = realloc(s->ptr, new_len+1);
  if (s->ptr == NULL) {

    fprintf(stderr, "realloc() failed\n");
    exit(EXIT_FAILURE);
  }
  memcpy(s->ptr+s->len, ptr, size*nmemb);
  s->ptr[new_len] = '\0';
  s->len = new_len;

  return size*nmemb;
}

int main(void)
{
  CURL *curl;
  CURLcode res;

  curl = curl_easy_init();
  if(curl) {
    struct string s;
    init_string(&s);

    curl_easy_setopt(curl, CURLOPT_URL, "https://stream.twitter.com/1/statuses/sample.json");
    curl_easy_setopt(curl, CURLOPT_USERPWD, "neilmarion:password_here");
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writefunc);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, &s); 
    res = curl_easy_perform(curl);

    printf("%s\n", s.ptr);
    free(s.ptr);

    /* always cleanup */
    curl_easy_cleanup(curl);
  }
  return 0;
}

Answer 1

OK, I did some testing and it appears that URL just keeps sending data, it doesn't seem to complete. I killed it off after 15M. But if you put print statements in your callback or use strace you can see its working properly. Your string s just keeps growing and growing.

So one solution would be to change you callback to print and re-initialse s once it reaches a certain size. Otherwise it looks like the program will eventually run out of memory. So change your callback to be

size_t max_buffer = 10240;  // 10K
size_t writefunc(void *ptr, size_t size, size_t nmemb, struct string *s)
{
  size_t new_len = s->len + size*nmemb;

  s->ptr = realloc(s->ptr, new_len+1);
  if (s->ptr == NULL) {
    fprintf(stderr, "realloc() failed\n");
    exit(EXIT_FAILURE);
  }
  memcpy(s->ptr+s->len, ptr, size*nmemb);
  s->ptr[new_len] = '\0';
  s->len = new_len;
  // Begin newly added code
  if( s->len >= max_buffer )
  {
    printf("%s", s->ptr);
    fflush( stdout );
    free(s->ptr);
    initString( s );
  }
  // End newly added code
  return size*nmemb;
}

And still keep the print at the end. To dump the last bit and the trailing newline. Now that you have a buffered solution you could look at a more efficient implementation which doesn't need dynamically added memory.