Reading data from an open HTTP stream

Question

I am trying to use the .NET WebRequest/WebResponse classes to access the Twitter streaming API here "http://stream.twitter.com/spritzer.json".

I need to be able to open the connection and read data incrementally from the open connection.

Currently, when I call WebRequest.GetResponse method, it blocks until the entire response is downloaded. I know there is a BeginGetResponse method, but this will just do the same thing on a background thread. I need to get access to the response stream while the download is still happening. This just does not seem possible to me with these classes.

There is a specific comment about this in the Twitter documentation:

"Please note that some HTTP client libraries only return the response body after the connection has been closed by the server. These clients will not work for accessing the Streaming API. You must use an HTTP client that will return response data incrementally. Most robust HTTP client libraries will provide this functionality. The Apache HttpClient will handle this use case, for example."

They point to the Appache HttpClient, but that doesn't help much because I need to use .NET.

Any ideas whether this is possible with WebRequest/WebResponse, or do I have to go for lower level networking classes? Maybe there are other libraries that will allow me to do this?

Thx Allen

score 16 · Answer 1 · edited Feb 26 '20 at 18:27

I ended up using a TcpClient, which works fine. Would still be interested to know if this is possible with WebRequest/WebResponse though. Here is my code in case anybody is interested:

using (TcpClient client = new TcpClient())
{

    string requestString = "GET /spritzer.json HTTP/1.1\r\n";
    requestString += "Authorization: " + token + "\r\n";
    requestString += "Host: stream.twitter.com\r\n";
    requestString += "Connection: keep-alive\r\n";
    requestString += "\r\n";

    client.Connect("stream.twitter.com", 80);

    using (NetworkStream stream = client.GetStream())
    {
        // Send the request.
        StreamWriter writer = new StreamWriter(stream);
        writer.Write(requestString);
        writer.Flush();

        // Process the response.
        StreamReader rdr = new StreamReader(stream);

        while (!rdr.EndOfStream)
        {
            Console.WriteLine(rdr.ReadLine());
        }
    }
}

Yes, HTTP is line-oriented protocol. You need to separate commands by \r\n. — Viet, Jul 05 '09 at 09:17
Is the stream returned by client.GetStream() actually streamed from the network stack or is it just a cached memory stream? — Agile Jedi, Jun 01 '11 at 03:15

Darin Dimitrov · Answer 2 · 2009-07-08T16:53:31.113

BeginGetResponse is the method you need. It allows you to read the response stream incrementally:

class Program
{
    static void Main(string[] args)
    {
        WebRequest request = WebRequest.Create("http://stream.twitter.com/spritzer.json");
        request.Credentials = new NetworkCredential("username", "password");
        request.BeginGetResponse(ar => 
        {
            var req = (WebRequest)ar.AsyncState;
            // TODO: Add exception handling: EndGetResponse could throw
            using (var response = req.EndGetResponse(ar))
            using (var reader = new StreamReader(response.GetResponseStream()))
            {
                // This loop goes as long as twitter is streaming
                while (!reader.EndOfStream)
                {
                    Console.WriteLine(reader.ReadLine());
                }
            }
        }, request);

        // Press Enter to stop program
        Console.ReadLine();
    }
}

Or if you feel more comfortable with WebClient (I personnally prefer it over WebRequest):

using (var client = new WebClient())
{
    client.Credentials = new NetworkCredential("username", "password");
    client.OpenReadCompleted += (sender, e) =>
    {
        using (var reader = new StreamReader(e.Result))
        {
            while (!reader.EndOfStream)
            {
                Console.WriteLine(reader.ReadLine());
            }
        }
    };
    client.OpenReadAsync(new Uri("http://stream.twitter.com/spritzer.json"));
}
Console.ReadLine();

I didn't try this because I assumed the req.EndGetResponse would block in exactly the same way req.GetResponse does, leaving me with the same problem. I will give it a shot. — user108687, Jul 08 '09 at 15:59
I tried this without any luck. The stream returned by .GetResponseStream() is a MemoryStream containing the entire contents of the response. — Jeb, Sep 01 '10 at 19:45
@Darin Dimitrov, How do you deal with timeouts if there are long periods of time between data coming through on the stream? — Brian Behm, May 11 '16 at 18:53
Thanks the WebClient example worked for me when connecting to Twitter filtered stream endpoint `https://api.twitter.com/2/tweets/search/stream`. I only needed to replace the Credentials line with `client.Headers[HttpRequestHeader.Authorization] = $"Bearer {settings.Twitter.BearerToken}";` and of course the Uri part. — Jan, Dec 18 '21 at 22:21

MikeB · Answer 3 · 2023-05-24T23:47:36.587

1

I think the modern way to do this is here:

var client = new HttpClient();
using var stream = await client.GetStreamAsync("http://stream.twitter.com/spritzer.json");

edited May 24 '23 at 23:47

answered May 23 '23 at 02:31

MikeB

1,452
14
28

score 0 · Answer 4 · answered Jul 04 '09 at 09:36

0

Have you tried WebRequest.BeginGetRequestStream() ?

Or something like this:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create (http://www.twitter.com );
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream()); 

string str = reader.ReadLine();
while(str != null)
{
   Console.WriteLine(str);
   str = reader.ReadLine();
}

answered Jul 04 '09 at 09:36

flesh

23,725
24
80
97

I can't use WebRequest.BeginGetRequestStream because I am not trying to write data asynchronously to the server. Also, in your example you call GetResponse. It is at this point that the WebRequest is blocking while it downloads the server response. But the connection never closes because it is an effectively endless stream of data. – user108687 Jul 04 '09 at 10:00

Reading data from an open HTTP stream

4 Answers4

Linked