How can I get HttpWebRequest to behave as synchronously as possible?

Question

Summary: Is this even possible?

unit test function on ui thread:
- creates a background thread
- starts the thread
- waits for it to complete (function does not exit until it completes!)

background thread:
- opens an http stream 
- reads a url from the web
- terminates

My suspicion: The framework asynchronously puts the result onto some internal message queue and thus the response callback will never be called until the ui thread's stack unwinds and goes to some ui thread function to pump the stack.

The full story:

I am porting an app that requires creating a stream from various sources, one of them being from a simple http url. I am doing this on a background thread, and ideally I would like it to behave 100% synchronously, just block when needed (this is ok since it's on a background thread).

But it seems the framework is a bit mickey mouse in that it assumes you will be doing the request on the ui thread and so it will shield the coder from having to create a background thread to do the asynch operation. But I may be missing something.

I have stumbled across the following article: http://pieterderycke.wordpress.com/2011/05/23/adding-synchronous-methods-to-webrequest-on-windows-phone-7/, which suggests a solution to make the http web request synchronous. But as it is implemented, I get a ProtocolViolationException. I have since made a modification to the code to use BeginGetResponse() instead of BeginGetRequestStream(), and this seems to no longer cause the exception.

But it seems that the background thread now blocks indefinitely. On my ui thread I loop, doing a Thread.Sleep(10) since I am in a unit test function, waiting for my callback to get called. Is it possible that the callback will not be called until unit test function returns and the ui thread has a chance to pump messages? If so, any way I can force it to pump so that I can continue where I left off in the unit test routine?

At the bottom of the article mentioned above, a comment is made "If you test your code, you will find that it deadlocks if you execute it on the UI thread." but I am executing it on a background thread, so should be ok right?

The msdn docs only show you how to do asynch calls. And they also mention that "The BeginGetResponse method requires some synchronous setup tasks to complete" ... "typically several seconds" ... but "can take 60 seconds or more". This sounds pretty bad to be executed on a ui thread. http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.begingetresponse.aspx

Please help!

Here is my code:

using System.Net;
using System.Threading;
using System;
using System.IO;

namespace Blah
{
  // http://pieterderycke.wordpress.com/2011/05/23/adding-synchronous-methods-to-webrequest-on-windows-phone-7/
  // Creates synchronous web requests.
  // Must not be called on UI threads.

  public static class WebRequestExtensions
  {
    public static Stream GetRequestStream(this WebRequest request)
    {
      AutoResetEvent autoResetEvent = new AutoResetEvent(false);

      IAsyncResult asyncResult = null;
      {
        // http://stackoverflow.com/questions/253549/how-do-i-use-httpwebrequest-with-get-method

        if (request.Method == "GET")
        {
          asyncResult = request.BeginGetResponse(
           r => autoResetEvent.Set(), null);
        }
        else
        {
          asyncResult = request.BeginGetRequestStream(
           r => autoResetEvent.Set(), null);
        }
      }

      // Wait until the call is finished
      autoResetEvent.WaitOne();

      return request.EndGetRequestStream(asyncResult);
    }
  }
}

I've also recently stumbled across http://www.eggheadcafe.com/tutorials/aspnet/91f69224-3da5-4959-9901-c5c717c9b184/making-silverlight-emulate-synchronous-requests.aspx, but this exhibits the same problem. Seems that I don't get my callback until the ui thread returns up the stack... I can smell some sort of a framework message queue there somewhere, am I correct?

Thanks

The code looks fine to me. I think the ProtocolViolationException is telling you something! What URL are you using? what was the full exception message? — ColinE, Sep 04 '11 at 06:41
Thanks, but like I said the ProtocolViolationException has gone away since I replaced request.BeginGetRequestStream() with request.BeginGetResponse(). — swinefeaster, Sep 04 '11 at 07:04

Richard Szalay · Accepted Answer · 2011-09-04T19:26:06.567

3

As HTTP response processing utilises the UI thread, blocking it will prevent the request from completing.

Assuming you are using the Silverlight Unit Testing Framework bits, you can mark your test as [Asynchronous]:

using System;
using System.Net;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.Silverlight.Testing;

[TestClass]
public class WebRequestsTests : WorkItemTest
{
    [TestMethod, Asynchronous]
    public void TestWebRequest()
    {
        var webRequest = WebRequest.CreateHttp("http://www.stackoverflow.com");

        webRequest.BeginGetResponse(result =>
        {
            EnqueueCallback(() =>
            {
                WebResponse response = webRequest.EndGetResponse(result);

                // process response 

                TestComplete(); // async test complete 
            });
        }, null);
    }
}

Alternatively, if you're looking to use Rx (which I do, personally), I recently did a blog post on how to make Rx-based asynchronous tests cleaner when using the SL testing framework.

edited Sep 04 '11 at 19:26

answered Sep 04 '11 at 09:10

Richard Szalay

83,269
19
178
237

@swinefeaster - The first line of my answer is the direct response to your question (that it's not possible and why). – Richard Szalay Sep 04 '11 at 14:02
Interesting, this might work, but I will need to generalize this solution to work with my engine. I am using the test framework from http://www.jeff.wilcox.name/2011/06/updated-ut-mango-bits/, and the [Asynchronous] tag doesn't seem to be supported (I get a compiler error). – swinefeaster Sep 04 '11 at 14:51
@swinefeaster - It is indeed part of the framework, but it's in the `Microsoft.Silverlight.Testing` namespace. – Richard Szalay Sep 04 '11 at 15:49
So any ideas how I can get your solution to compile? – swinefeaster Sep 04 '11 at 19:17
@swinefeaster - Updated to compile, including required using statements. – Richard Szalay Sep 04 '11 at 19:26
Looks like this approach is not supported in WP7. Microsoft.VisualStudio.TestTools.UnitTesting is not supported. – swinefeaster Sep 05 '11 at 06:46
@swinefeaster - I just ran some asynchronous tests in the emulator 30 seconds ago, so I guarantee it's supported in WP7. `Assert` is in `Microsoft.VisualStudio.TestTools.UnitTesting`, so if it wasn't supported there wouldn't be much point to the testing framework. – Richard Szalay Sep 05 '11 at 07:34
@swinefeaster - Make sure you've referenced both the `Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll` and `Microsoft.Silverlight.Testing.dll` assemblies, which both ship with the link you posted. – Richard Szalay Sep 05 '11 at 07:44
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/3189/discussion-between-swinefeaster-and-richard-szalay) – swinefeaster Sep 05 '11 at 19:03
Ok so this seems to be the best answer to get the unit testing stuff to work. My current implementation will likely work in the final build without any changes. I will have to start a new question as I'm not sure how to run the unit tests in wp7 without a runner such as nunit. – swinefeaster Sep 08 '11 at 04:44
Hey Richard, if I wanted to use the same pattern as above in the unit test, but instead of the unit test waiting on the http request, it would wait on a user-defined async callback that is triggered when my thread is done, how would I go about doing this? What interfaces would I need to implement? Thanks – swinefeaster Sep 10 '11 at 03:33
Ok I've started a new question for this: http://stackoverflow.com/questions/7370025/how-can-i-implement-my-own-async-callback-in-c-wp7-silverlight – swinefeaster Sep 10 '11 at 06:24

score 2 · Answer 2 · answered Sep 04 '11 at 20:50

While I'm not advocating moving stuff to the UI thread, which is what you're probably going to end up with, there are times when porting synchronous code to the Phone that it would be nice to have a synchronous version of GetResponse. I wrote these extensions methods for a library I maintain that's compatible with multiple platforms.

score 1 · Answer 3 · edited Sep 04 '11 at 08:48

1

Why ask for a bad solution to the wrong problem? If you need continuations on asynchronous requests, you should simply use the Task Parallel Library.

You can find a version of TPL for Windows Phone on NuGet: Task Parallel Library for Silverlight.

This allows continuations, which essentially would give the same effect as blocking your thread.

(Robert McLaws, creator of the library) NuGet is a system that makes it easy to manage dependencies. Just go to http://nuget.org/, click the link to install it, then open the Package Manager Console and type "Install-Package System.Threading.Tasks", and it will automatically install the right version for your project. I am not currently making the source available, although I might sometime in the future. I need to check with the Mono guys, as the code comes mostly from Mono.

edited Sep 04 '11 at 08:48

Robert McLaws

2,258
2
19
22

answered Sep 04 '11 at 07:24

Claus Jørgensen

25,882
9
87
150

I don't see a download link there to the source files... ? – swinefeaster Sep 04 '11 at 07:55
Robert added his comment in a edit to my post (since he somehow couldn't write it as a comment) – Claus Jørgensen Sep 04 '11 at 08:49
Can you show an example of how I can use this library to synchronously get the contents of a web page and return its contents as a string? Thanks – swinefeaster Sep 04 '11 at 14:14
Again, it's not synchronously in the traditional sense. The point is that you *continue* after the asynchronous operation have finished, using callbacks. Which essentially is the same if you code for it. – Claus Jørgensen Sep 04 '11 at 14:24

score 1 · Answer 4 · answered Sep 04 '11 at 07:43

1

you could also use Rx (that ships with the library) to do this.

http://dotnet.dzone.com/articles/5-minute-sample-fromasyncpattern

answered Sep 04 '11 at 07:43

Didier Caron

570
3
9

Does this actually make sync calls out of async calls? At first glance it seems that it just helps with syntax but underneath it still does async... – swinefeaster Sep 04 '11 at 07:58
Yeah, but like with TPL, Rx uses continuations, meaning you can just execute the code when the (async) request is done, instead of having to block the thread. – Claus Jørgensen Sep 04 '11 at 08:53
it does, it actually should have been a remark underneat the answer of Claus. You can do the things you desire with Rx though.. i would always prefer a publish and subscribe pattern over an polling one – Didier Caron Sep 04 '11 at 08:54
I'd rather someone upvotes your answer, since you provided a guide on how to implement it ;-) – Claus Jørgensen Sep 04 '11 at 12:22
Seems the root problem here is that NUnit doesn't support Asynchronous tests. I believe your solution would not address this, because the test method would exit before the asynch response is received. – swinefeaster Sep 06 '11 at 03:13

How can I get HttpWebRequest to behave as synchronously as possible?

4 Answers4

Linked