HttpSendRequest - POST data not supporting Unicode

Question

I'm working on making a C++ agent that will post information (such as the system hostname) back to a central server using HttpSendRequest(). One of the pieces of information that I want it to post back is the OS. I created the following function that will obtain the system hostname.

wstring getOS()
{
     HKEY key;
     RegOpenKeyEx(HKEY_LOCAL_MACHINE, L"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion", 0, KEY_QUERY_VALUE, &key); // Obtains Registry handle

     DWORD type;
     wchar_t buffer[MAX_PATH]; // MAX_PATH = 260 - The system hostname should never exceed this value
     DWORD size = sizeof(buffer);
     RegQueryValueEx(key, L"ProductName", NULL, &type, (LPBYTE)&buffer, &size); // Queries Registry key - stores value in "buffer"
     wstring os(buffer); // Converts from C-style character array to wstring
     return os; // Returns wstring to caller
}

This function will obtain the OS using the Registry and store it as a wstring. I then want to pass the returned "os" wstring to the following post() function, but I noticed that you must use a string instead of a wstring for the HTTP POST data. Below is the code for my post() function:

void post()
{
     HINTERNET hInternetOpen = InternetOpen(userAgent.c_str(), INTERNET_OPEN_TYPE_PROXY, L"http://127.0.0.1:9999", NULL, 0);
     HINTERNET hInternetConnect = InternetConnect(hInternetOpen, host.c_str(), INTERNET_DEFAULT_HTTP_PORT, NULL, NULL, INTERNET_SERVICE_HTTP, 0, 0);
     HINTERNET hHttpOpenRequest = HttpOpenRequest(hInternetConnect, L"POST", file.c_str(), NULL, NULL, NULL, 0, 0);

     wstring headers = L"Content-Type: application/x-www-form-urlencoded"; // Content-Type is necessary to POST

     string postData = "os="; // Why does this have to be a string and not a wstring?

     HttpSendRequest(hHttpOpenRequest, headers.c_str(), headers.length(), (LPVOID)postData.c_str(), postData.size());

     InternetCloseHandle(hInternetOpen);
     InternetCloseHandle(hInternetConnect);
     InternetCloseHandle(hHttpOpenRequest);
}

If I try to make "postData" a wstring, I get something that looks like the image below:

Can someone shed some light onto the easiest way to include a wstring as the POST data?

If I try to use a wstring instead of a string, it messes up the POST data (see attached screenshot). — t3ntman, Apr 21 '16 at 15:51
what do you want sent on the wire? utf8? see here http://stackoverflow.com/questions/148403/utf8-to-from-wide-char-conversion-in-stl — pm100, Apr 21 '16 at 16:12
`size()` returns the number of characters in the string not the amount of memory they occupy. If you're calling `size()` on a `wstring` like you are on the `string` currently used in your code you're doing it wrong. — Captain Obvlious, Apr 21 '16 at 16:34
@CaptainOblivious thank you, I have changed it to use .length() instead of .size(). — t3ntman, Apr 21 '16 at 19:18
@pm100 yes, UTF-8 so that I can target different languages. I will look at that link and see if that helps. Thanks. — t3ntman, Apr 21 '16 at 19:18
So I think I'm making progress. From the HttpSendRequest() MSDN page: "WideCharToMultiByte functions with a Codepage parameter set to 28591 to map between ANSI characters and UTF-16LE characters." Any idea on how to use the WideCharToMultiByte() function? — t3ntman, Apr 21 '16 at 20:11

Remy Lebeau · Accepted Answer · 2016-04-21T20:29:02.533

HttpSendRequest() only knows about raw bytes, not strings. You can send UTF-16 data using a std::wstring, but you have to tell the server that you are sending UTF-16, via a charset attribute in the Content-Type header.

wstring headers = L"Content-Type: application/x-www-form-urlencoded; charset=utf-16";

// TODO: don't forget to URL-encode the value from getOS() to
// escape reserved characters, including '=' and '&'...
wstring postData = L"os=" + getOS();

HttpSendRequest(hHttpOpenRequest, headers.c_str(), headers.length(),
    postData.c_str(), postData.length() * sizeof(wchar_t));

Note the use of sizeof(wchar_t) above. In your screenshot, your sniffer is showing the raw data, and the data it shows is what UTF-16 would look like, but you see only half of your wstring data because you are setting the dwOptionalLength parameter of HttpSendRequest() to a character count (7) instead of a byte count (14):

dwOptionalLength [in]
The size of the optional data, in bytes. This parameter can be zero if there is no optional data to send.

When you use std::string, the character count and the byte count are the same value.

What you really should be sending is UTF-8 instead of UTF-16, eg:

string Utf8Encode(const wstring &wstr)
{
    // NOTE: C++11 has built-in support for converting between
    // UTF-8 and UTF-16.  See the std::wstring_convert class...
    /*
    wstring_convert<codecvt_utf8_utf16<wchar_t>> conv;
    return conv.to_bytes(wstr);
    */

    string out;
    int len = WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), wstr.length(), NULL, 0, NULL, NULL);
    if (len > 0)
    {
        out.resize(len);
        WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), wstr.length(), &out[0], len, NULL, NULL);
    }
    return out;
}

wstring headers = L"Content-Type: application/x-www-form-urlencoded; charset=utf-8";

// TODO: don't forget to URL-encode the value from getOS() to
// escape reserved characters, including '=' and '&'...
string postData = "os=" + Utf8Encode(getOS());

HttpSendRequest(hHttpOpenRequest, headers.c_str(), headers.length(),
    postData.c_str(), postData.size());

You are the man! This worked and should be the accepted answer. I would upvote it if I had enough reputation :/ Thank you so very much! — t3ntman, Apr 21 '16 at 20:39
I just clicked the check mark to do so. Please let me know if that didn't work. Thanks again. — t3ntman, Apr 21 '16 at 23:55

HttpSendRequest - POST data not supporting Unicode

1 Answers1