How do I use the winhttp api with "transfer-encoding: chunked"

Question

I'm trying to send some data to a web service which requires the "Transfer-encoding: chunked" header. It works fine with a normal POST request. But as soon as I add the header, I always get:

The content could not be delivered due to the following condition: Received invalid request from client

This is the part where the request is sent:

std::vector<std::wstring> m_headers;
m_headers.push_back(TEXT("Transfer-encoding: chunked"));
std::wstring m_verb(TEXT("POST"));
std::vector<unsigned __int8> m_payload;

HINTERNET m_connectionHandle = WinHttpConnect(m_http->getSessionHandle(), hostName.c_str(), m_urlParts.nPort, 0);
if (!m_connectionHandle) {
    std::cout << "InternetConnect failed: " << GetLastError() << std::endl;
    return;
}

__int32 requestFlags = WINHTTP_FLAG_SECURE | WINHTTP_FLAG_REFRESH;
HINTERNET m_requestHandle = WinHttpOpenRequest(m_connectionHandle, m_verb.c_str(), (path + extra).c_str(), NULL, WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, requestFlags);
if(!m_requestHandle) {
    std::cout << "HttpOpenRequest failed: " << GetLastError() << std::endl;
    return;
}

for(auto header : m_headers) {
    if(!WinHttpAddRequestHeaders(m_requestHandle, (header + TEXT("\r\n")).c_str(), -1, WINHTTP_ADDREQ_FLAG_ADD)) {
        std::cout << "WinHttpAddRequestHeaders failed: " << GetLastError() << std::endl;
        return;
    }
}

if(!WinHttpSendRequest(m_requestHandle, WINHTTP_NO_ADDITIONAL_HEADERS, 0, WINHTTP_NO_REQUEST_DATA, 0, WINHTTP_IGNORE_REQUEST_TOTAL_LENGTH, (DWORD_PTR)this)) {
    std::cout << "HttpSendRequest failed: " << GetLastError() << std::endl;
    return;
}

unsigned chunkSize = 1024;
unsigned chunkCount = m_payload.size() / chunkSize;
char chunksizeString[128];
for (unsigned i = 0; i <= chunkCount; i++) {
    unsigned actualChunkSize = std::min<unsigned>(chunkSize, m_payload.size() - i * chunkSize);
    sprintf_s(chunksizeString, "%d\r\n", actualChunkSize);
    if (!WinHttpWriteData(m_requestHandle, chunksizeString, strlen(chunksizeString), (LPDWORD)&m_totalBytesWritten)) {
        std::cout << "HttpWriteData failed: " << GetLastError() << std::endl;
        return;
    }
    if (!WinHttpWriteData(m_requestHandle, m_payload.data() + i * chunkSize, actualChunkSize, (LPDWORD)&m_totalBytesWritten)) {
        std::cout << "HttpWriteData failed: " << GetLastError() << std::endl;
        return;
    }
}

// terminate chunked transfer
if (!WinHttpWriteData(m_requestHandle, "0\r\n", strlen("0\r\n"), (LPDWORD)&m_totalBytesWritten)) {
    std::cout << "HttpWriteData failed: " << GetLastError() << std::endl;
    return;
}

if(!WinHttpReceiveResponse(m_requestHandle, NULL)) {
    std::wcout << "HttpReceiveResponse failed: " << GetLastError() << std::endl;
    return;
}

I had to copy it from different files, so I hope I got all the important variable definitions. Right now I only use it synchronously since I thought it easier to debug.

As it works with normal POST requests (where I just use WinHttpSendRequest with the payload) I'm guessing it must have to do with the way I use WinHttpSendRequest & WinHttpWriteData, I just don't see how else it should be used.

Any help is appreciated!

When passing your `vector` to `WinHttpWriteData()`, you are passing the wrong pointer to the `lpBuffer` parameter. You need to change `&m_payload` to either `&m_payload[0]` or `m_payload.data()` — Remy Lebeau, Oct 23 '17 at 19:25

score 2 · Answer 1 · answered Oct 24 '17 at 08:57

2

You need to split data into chunks manually like this:

int chunkSize = 512;       // can be anything
char chunkSizeString[128]; // large enough string buffer
for (int i=0; i<chunksCount; ++i) {
    int actualChunkSize = chunkSize; // may be less when passing the last chunk of data (if that's not a multiple of chunkSize)
    sprintf(chunkSizeString, "%d\r\n", actualChunkSize);
    WinHttpWriteData(m_requestHandle, chunkSizeString, strlen(chunkSizeString), (LPDWORD)&m_totalBytesWritten);
    WinHttpWriteData(m_requestHandle, m_payload.data() + i*chunkSize, actualChunkSize, (LPDWORD)&m_totalBytesWritten);
}
WinHttpWriteData(m_requestHandle, "0\r\n", strlen("0\r\n"), (LPDWORD)&m_totalBytesWritten); // the last zero chunk, end of transmission

answered Oct 24 '17 at 08:57

Anton Malyshev

8,686
2
27
45

So I have to tell winhttp the actual size I'm sending as string beforehand in a seperate WriteData call while there is a parameter for the size? This seems weird to me (Not saying I doubt your advice, I just don't get why). Anyways it doesn't change my problem, still getting the "invalid request" response. I'm new to http stuff, is there a way to just display the actual request before sending it out? – sph Oct 24 '17 at 09:47
Show your modified code, maybe there are some errors left – Anton Malyshev Oct 24 '17 at 09:48
I edited it in above. Right now it actually fails most of the time in one of the WriteData calls with ERROR_WINHTTP_CONNECTION_ERROR. When it goes through though I get the same html response as before. – sph Oct 24 '17 at 10:05
Here is an error: `chunkCount = m_payload.size() / chunkSize;` - it's wrong when `m_payload.size()` if not multiple of `chunkSize` (need to +1); Also check that `m_totalBytesWritten` equals to the size of data passed. – Anton Malyshev Oct 24 '17 at 10:37
Your right that for semantics it should probably say +1, but I instead iterate until <= so I got every chunk. Should check against the bytes written though, true. But I did check while debugging! – sph Oct 24 '17 at 10:53
Also, as a side node, I should be able to send all the data in one chunk as I did before as long as I send the terminating WinHttpWriteData, right? – sph Oct 24 '17 at 10:57
Yes, but you need to specify chunk size before sending the chunk (what wasn't done before). You can try to compare your code with some working code, for example: https://github.com/libgit2/libgit2/blob/master/src/transports/winhttp.c – Anton Malyshev Oct 24 '17 at 11:04
Thanks for the link, I found it quite hard to find an example. I'll look into it! – sph Oct 24 '17 at 12:49

score 1 · Accepted Answer · answered Oct 24 '17 at 14:49

Thanks to the link provided by @anton-malyshev I was able to find the solution, I just replaced all calls to WinHttpWriteData above with this:

/* Chunk header */
char chunksizeString[64];
sprintf_s(chunksizeString, "%X\r\n", m_payload.size());
if (!WinHttpWriteData(m_requestHandle, chunksizeString, strlen(chunksizeString), (LPDWORD)&m_totalBytesWritten)) {
    std::wcout << "WinHttpWriteData chunk header failed: " << getHttpErrorMessage(GetLastError()) << std::endl;
    return;
}

/* Chunk body */
if (!WinHttpWriteData(m_requestHandle, m_payload.data(), m_payload.size(), (LPDWORD)&m_totalBytesWritten)) {
    std::wcout << "WinHttpWriteData chunk body failed: " << getHttpErrorMessage(GetLastError()) << std::endl;
    return;
}

/* Chunk footer */
if (!WinHttpWriteData(m_requestHandle, "\r\n", 2, (LPDWORD)&m_totalBytesWritten)) {
    std::wcout << "WinHttpWriteDatachunk footer failed: " << getHttpErrorMessage(GetLastError()) << std::endl;
    return;
}

/* Terminate chunk transfer */
if (!WinHttpWriteData(m_requestHandle, "0\r\n\r\n", 5, (LPDWORD)&m_totalBytesWritten)) {
    std::wcout << "WinHttpWriteData chunk termination failed: " << getHttpErrorMessage(GetLastError()) << std::endl;
    return;
}

How do I use the winhttp api with "transfer-encoding: chunked"

2 Answers2

Linked