Unit testing HTTP requests in c#

Question

I'm writing some code that calls a web service, reads back the response and does something with it. My code looks nominally like this:

string body = CreateHttpBody(regularExpression, strategy);

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_url);
request.Method = "POST";
request.ContentType = "text/plain; charset=utf-8";

using (Stream requestStream = request.GetRequestStream())
{
    requestStream.Write(Encoding.UTF8.GetBytes(body), 0, body.Length);
    requestStream.Flush();
}

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
    byte[] data = new byte[response.ContentLength];

    using (Stream stream = response.GetResponseStream())
    {
        int bytesRead = 0;

        while (bytesRead < data.Length)
        {
            bytesRead += stream.Read(data, bytesRead, data.Length - bytesRead);
        }
    }

    return ExtractResponse(Encoding.UTF8.GetString(data));
}

The only parts where I am actually doing any custom manipulation is in the ExtractResponse and CreateHttpBody methods. However it feels wrong to just unit test those methods, and hope that the rest of the code comes together correctly. Is there any way I can intercept the HTTP request and feed it mock data instead?

EDIT This information is now out of date. It is much easier to construct this kind of code using the System.Net.Http.HttpClient libraries.

You could set up a _(misbehaving, or whatever you want to test)_ webserver and then modify the `\Windows\System32\drivers\etc\hosts` file to "redirect" requests to that server. — ordag, Feb 01 '12 at 12:32
a small philosophical comment: What happens if that url is not available? i.e. is that url being mocked or served by your unit test or do you depend on some server that you assume is available? The latter case is not unit testing — yerlilbilgin, Feb 04 '19 at 21:12

score 22 · Accepted Answer · edited Aug 07 '12 at 12:25

In your code you can not intercept the calls to HttpWebRequest because you create the object in the same method. If you let another object create the HttpWebRequest, you can pass in a mock object and use that to test.

So instead of this:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_url);

Use this:

IHttpWebRequest request = this.WebRequestFactory.Create(_url);

In your unit test, you can pass in a WebRequestFactory which creates a mock object.

Furthermore, you can split of your stream reading code in a separate function:

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
    byte[] data = ReadStream(response.GetResponseStream());
    return ExtractResponse(Encoding.UTF8.GetString(data));
}

This makes it possible to test ReadStream() separately.

To do more of an integration test, you can set up your own HTTP server which returns test data, and pass the URL of that server to your method.

After experimenting with Moles and Moq in an attempt to mock the HttpWebRequest and HttpWebResponse objects, I have to conclude that it can't be done, making this the most effective approach. — Ceilingfish, Feb 01 '12 at 16:52

score 10 · Answer 2 · answered Jan 19 '16 at 00:47

If mocking out the HttpWebRequest and HttpWebResponse becomes too cumbersome, or if you ever need to test code in an acceptance test, where you are calling your code from the "outside", then creating a fake service is probably the best way to go.

I actually wrote an open source library called MockHttpServer to assist with this, making it super simple to mock out any external services that communicate over HTTP.

Here is an example of using it with RestSharp to call an API endpoint with it, but HttpWebRequest would work just as well.

using (new MockServer(3333, "/api/customer", (req, rsp, prm) => "Result Body"))
{
    var client = new RestClient("http://localhost:3333/");
    var result = client.Execute(new RestRequest("/api/customer", Method.GET));
}

There is a fairly detailed readme on the GitHub page that goes through all the options available for using it, and the library itself is available through NuGet.

@Ceilingfish that project looks cool, but seems like overkill for simply mocking a few responses. It's an entire web server library, which adds a bit of complexity. — Jeffrey Harmon, Jan 20 '16 at 15:59

score 6 · Answer 3 · answered Feb 01 '12 at 12:29

I would probably start by refactoring the code in order to make it more weakly coupled to an actual HTTP request. Right now this code seems to do quite a lot of things.

This could be done by introducing an abstraction:

public interface IDataRetriever
{
    public byte[] RetrieveData(byte[] request);
}

Now the class that you are trying to unit test could be decoupled from the actual HTTP request using the Inversion of Control design pattern:

public class ClassToTest
{
    private readonly IDataRetriever _dataRetriever;
    public Foo(IDataRetriever dataRetriever)
    {
        _dataRetriever = dataRetriever;
    }

    public string MethodToTest(string regularExpression, string strategy)
    {
        string body = CreateHttpBody(regularExpression, strategy);
        byte[] result = _dataRetriever.RetrieveData(Encoding.UTF8.GetBytes(body));
        return ExtractResponse(Encoding.UTF8.GetString(result));
    }
}

It is no longer the ClassToTest's responsibility to deal with an actual HTTP request. It is now decoupled. Testing the MethodToTest becomes a trivial task.

And the last part obviously is to have an implementation of the abstraction that we have introduced:

public class MyDataRetriever : IDataRetriever
{
    private readonly string _url;
    public MyDataRetriever(string url)
    {
        _url = url;
    }

    public byte[] RetrieveData(byte[] request)
    {
        using (var client = new WebClient())
        {
            client.Headers[HttpRequestHeader.ContentType] = "text/plain; charset=utf-8";
            return client.UploadData(_url, request);
        }
    }
}

You could then configure your favorite DI framework to inject a MyDataRetriever instance into the ClassToTest class constructor in your actual application.

I can see your point, however the class where this code resides is designed to be an HTTP data retrieval mechanism. I want to test that the code that I am using to handle the request is well formed, so if I abstract away the HTTP code, then I am abstracting away exactly the part that I want to test. — Ceilingfish, Feb 01 '12 at 15:28
@Ceilingfish, no, you definitely don't want to test the HttpWebRequest and HttpWebResponse classes. Those are already extensively tested by Microsoft. What you want to test is that your method provides correct input to some black box (which is provided to you and which is expected to work) and that it converts the output of it to some expected format. That's basically what your method does. Anyway, without this level of abstraction you cannot mock those classes and you are no longer doing unit tests but integration tests. — Darin Dimitrov, Feb 01 '12 at 15:40
True, I don't want to be testing those classes, I want to test the _usage_ of those classes. The code to send and receive an HTTP request is not hidden entirely, so it relies on my code being correct to successfully read the response from the request. — Ceilingfish, Feb 01 '12 at 16:00

score 2 · Answer 4 · answered Jan 19 '16 at 01:04

If you're happy to move to HttpClient (an official, portable, http client library), then I wrote a library a while back that may help called MockHttp. It provides a fluent API that allows you provide responses for requests matched using a range of attributes.

Unit testing HTTP requests in c#

4 Answers4

Linked

Related