Create http request using TcpClient

Question

I have created an empty asp.net web application where I have a simple aspx page:

protected void Page_Load(object sender, EventArgs e)
{
    Response.Write("Hello world");
}

when i goto http://localhost:2006/1.aspx I see a page that says "Hello world".

---

Ok so on c# if I do:

WebClient webClient = new WebClient() { Proxy = null };
var response2 = webClient.DownloadString("http://localhost:2006/1.aspx");

then response2 == "Hello world"

I need to achieve the same thing with a raw tcp connection

I am trying to achieve the same thing with a tcp connection and for some reason it does not work:

byte[] buf = new byte[1024];
string header = "GET http://localhost:2006/1.aspx HTTP/1.1\r\n" +
                "Host: localhost:2006\r\n" +
                "Connection: keep-alive\r\n" +
                "User-Agent: Mozilla/5.0\r\n" +
                "\r\n";

var client = new TcpClient("localhost", 2006);            

// send request
client.Client.Send(System.Text.Encoding.ASCII.GetBytes(header));

// get response
var i = client.Client.Receive(buf);
var response1 = System.Text.Encoding.UTF8.GetString(buf, 0, i);

here response1 != "Hello Wold". (note I use != meaning NOT equal)

In this example I get a bad request error.

---

I want to use a tcp connection for learning purposes. I dont understand why the second example does not work. My first reaction was maybe the headers are incorrect so what I did is I launched wireshark in order to see the headers send by my chrom browser. In fact the actual request sent by my browser when I goto http://localhost:2006/1.aspx is:

GET http://localhost:2006/1.aspx HTTP/1.1
Host: localhost:2006
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8

I have also tried using that request and when I do so I also get a Bad Request response! Why?

In other words I have replaced

string header = "GET http://localhost:2006/1.aspx HTTP/1.1\r\n" +
                "Host: localhost:2006\r\n" +
                "Connection: keep-alive\r\n" +
                "User-Agent: Mozilla/5.0\r\n" +
                "\r\n";

FOR

string header = "GET http://localhost:2006/1.aspx HTTP/1.1\r\n" +
        "Host: localhost:2006\r\n" +
        "Connection: keep-alive\r\n" +
        "Cache-Control: max-age=0\r\n" +
        "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n" +
        "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36\r\n" +
        "Accept-Encoding: gzip,deflate,sdch\r\n" +
        "Accept-Language: en-US,en;q=0.8" +
        "\r\n\r\n";

and it still does not work.

Answer 1

Here is my version. Works fine

    private static async Task<string> HttpRequestAsync()
    {
        string result = string.Empty;

        using (var tcp = new TcpClient("www.bing.com", 80))
        using (var stream = tcp.GetStream())
        {
            tcp.SendTimeout = 500;
            tcp.ReceiveTimeout = 1000;
            // Send request headers
            var builder = new StringBuilder();
            builder.AppendLine("GET /?scope=images&nr=1 HTTP/1.1");
            builder.AppendLine("Host: www.bing.com");
            //builder.AppendLine("Content-Length: " + data.Length);   // only for POST request
            builder.AppendLine("Connection: close");
            builder.AppendLine();
            var header = Encoding.ASCII.GetBytes(builder.ToString());
            await stream.WriteAsync(header, 0, header.Length);

            // Send payload data if you are POST request
            //await stream.WriteAsync(data, 0, data.Length);

            // receive data
            using (var memory = new MemoryStream())
            {
                await stream.CopyToAsync(memory);
                memory.Position = 0;
                var data = memory.ToArray();

                var index = BinaryMatch(data, Encoding.ASCII.GetBytes("\r\n\r\n")) + 4;
                var headers = Encoding.ASCII.GetString(data, 0, index);
                memory.Position = index;

                if (headers.IndexOf("Content-Encoding: gzip") > 0)
                {
                    using (GZipStream decompressionStream = new GZipStream(memory, CompressionMode.Decompress))
                    using (var decompressedMemory = new MemoryStream())
                    {
                        decompressionStream.CopyTo(decompressedMemory);
                        decompressedMemory.Position = 0;
                        result = Encoding.UTF8.GetString(decompressedMemory.ToArray());
                    }
                }
                else
                {
                    result = Encoding.UTF8.GetString(data, index, data.Length - index);
                    //result = Encoding.GetEncoding("gbk").GetString(data, index, data.Length - index);
                }
            }

            //Debug.WriteLine(result);
            return result;
        }
    }

    private static int BinaryMatch(byte[] input, byte[] pattern)
    {
        int sLen = input.Length - pattern.Length + 1;
        for (int i = 0; i < sLen; ++i)
        {
            bool match = true;
            for (int j = 0; j < pattern.Length; ++j)
            {
                if (input[i + j] != pattern[j])
                {
                    match = false;
                    break;
                }
            }
            if (match)
            {
                return i;
            }
        }
        return -1;
    }

Answer 2

I had to do this today because I was dealing with an HTTP server that returned invalid HTTP responses and it could not be fixed. HttpClient kept throwing exceptions because of the invalid responses, so I had to go down one level in the network stack and send the HTTP message using TCP.

I got the original code from here but it needed fixing, and refactoring, so here's my version of it:

using System.Net.Security;
using System.Net.Sockets;
using System.Security.Authentication;
using System.Security.Cryptography.X509Certificates;

namespace Demo
{
    public static class TcpClientForHttp
    {
        public static Task<MemoryStream> SendHttpRequestAsync(
            Uri uri,
            HttpMethod httpMethod,
            Version httpVersion,
            CancellationToken cancellationToken)
        {
            string strHttpRequest = $"{httpMethod} {uri.PathAndQuery} HTTP/{httpVersion}\r\n";
            strHttpRequest += $"Host: {uri.Host}:{uri.Port}\r\n";
            // Any other HTTP headers can be added here ....
            strHttpRequest += "\r\n";

            return SendRequestAsync(uri, strHttpRequest, cancellationToken);
        }

        private static async Task<MemoryStream> SendRequestAsync(Uri uri, string request, CancellationToken token)
        {
            bool isHttps = uri.Scheme == Uri.UriSchemeHttps;

            using var tcpClient = new TcpClient();
            await tcpClient.ConnectAsync(uri.Host, uri.Port, token);
            await using NetworkStream ns = tcpClient.GetStream();

            var resultStream = new MemoryStream();

            if (isHttps)
            {
                await using var ssl = new SslStream(ns, false, ValidateServerCertificate, null);
                await ssl.AuthenticateAsClientAsync(new SslClientAuthenticationOptions
                    {
                        TargetHost = uri.Host,
                        ClientCertificates = null,
                        EnabledSslProtocols = SslProtocols.None,
                        CertificateRevocationCheckMode = X509RevocationMode.NoCheck
                    },
                    token);

                await using var sslWriter = new StreamWriter(ssl);
                await sslWriter.WriteAsync(request);
                await sslWriter.FlushAsync();

                await ssl.CopyToAsync(resultStream, token);
            }
            else
            {
                // Normal HTTP
                await using var nsWriter = new StreamWriter(ns);
                await nsWriter.WriteAsync(request);
                await nsWriter.FlushAsync();

                await ns.CopyToAsync(resultStream, token);
            }

            resultStream.Position = 0;
            return resultStream;
        }

        private static bool ValidateServerCertificate(
            object sender,
            X509Certificate? certificate,
            X509Chain? chain,
            SslPolicyErrors sslPolicyErrors)
        {
            return true; // Accept all certs
        }
    }
}

I removed a lot of the headers that I didn't need, made the code async, and returned a stream instead of a string. I didn't test the SSL code at all because I didn't need it.

Bonus tip: You can use a tool like Hercules to create a test TCP Server. You can even connect to it using HttpClient and send some garbage (I did that to reproduce the exception locally), but if you do that, remember to add <LF> to the end of the message so that HttpClient knows to stop the connection. With TcpClient you'd have to just actually close the server connection.