4

I´m trying to use the boost beast http library for an HTTP Client. It´s working without any issues when I´m using a simulated server, however when I try connecting to the real server, boost::beast::http::read throws an exception saying "partial message".

I´ve been working on this issue for a couple days now but I can´t figure out why. Until now I´ve been using a different http client library and the server communication has been working without any similar issues.

I´d be grateful for any kind of idea or hint as to why this is happening and why it doesn't seem to be an issue when using a different library.

sehe
  • 374,641
  • 47
  • 450
  • 633
abcd
  • 51
  • 1
  • 2

1 Answers1

3

boost::beast::http::read throws an exception saying "partial message".

This happens because the message being parsed wasn't complete. A typical reason for it is when the content-length header is wrong, or the sender abandons the connection prematurely. E.g.:

Live On Compiler Explorer

This is what http::[async_]read ends up doing under the hood, but without the network related stuff:

#include <iostream>
#include <iomanip>
#include <string_view>
#include <boost/beast/http.hpp>

int main() {
    using namespace boost::beast::http;
    using boost::asio::buffer;

    for (std::string_view buf : {
            "GET / HTTP/1.1\r\n", // incomplete headers
            "GET / HTTP/1.1\r\nHost: example.com\r\nContent-Length: 0\r\n\r\ntrailing data",
            "GET / HTTP/1.1\r\nHost: example.com\r\nContent-Length: 42\r\n\r\nshort",
        })
    {
        //std::cout << std::quoted(test) << "\n";
        std::cout << "---------------------" << "\n";

        request_parser<string_body> parser;
        boost::system::error_code ec;

        size_t n = parser.put(buffer(buf), ec);
        if (n && !ec && !parser.is_done()) {
            buf.remove_prefix(n);
            n = parser.put(buffer(buf), ec); // body
        }
        if (!ec)
            parser.put_eof(ec);
        buf.remove_prefix(n);

        std::cout
            << (parser.is_header_done()?"headers ok":"incomplete headers")
            << " / " << (parser.is_done()?"done":"not done")
            << " / " << ec.message() << "\n";
        if (parser.is_header_done() && !parser.is_done())
            std::cout << parser.content_length_remaining().value_or(0) << " more content bytes expected\n";

        if (!buf.empty())
            std::cout << "Remaining buffer: " << std::quoted(buf) << "\n";
    }
}

Prints

---------------------
incomplete headers / not done / need more
---------------------
headers ok / done / Success
Remaining buffer: "trailing data"
---------------------
headers ok / not done / partial message
37 more content bytes expected

If you're not passing error_code to your calls they will throw the exception system_error with the same code, which is exactly what you see.

Side Note

If another library doesn't have this "problem" there are two options:

  • the library is sloppy (i.e. bad)
  • you're using it wrong (maybe you're not checking for errors)
sehe
  • 374,641
  • 47
  • 450
  • 633
  • [Made the demo diagnostics a bit more detailed](https://stackoverflow.com/posts/66141381/revisions), Luckily you don't have to code any of that. Just `http::read(s, buf, req)` and handle your exceptions :) – sehe Feb 10 '21 at 17:16
  • Hi, thanks for the help. I did also try passing the error code and basically moving along with the Rest of the method. To my understanding I should at least be getting whatever has been read until the eof was reached, however according to the logs, the response is empty. While it is possible, that it´s the library´s fault (after all there´s a reason why I´m trying to use beast), the library should also react to these errors and we are checking for them. – abcd Feb 12 '21 at 11:29
  • BTW sorry if I´m switching a lot between I and we. I´m working for a large company and some things are done by me, while others are being handled by other people in my team – abcd Feb 12 '21 at 11:36
  • The good news is that the above shows you exactly how to use the parser to even get the result if it was incomplete. Several approaches exists. If the headers were done bu the body is missing data, you could pad it (see `content_length_remaining`), or you could roll with the body you got and log a a warning/error that an incomplete request was received. – sehe Feb 12 '21 at 13:02
  • I can´t find where in your example it shows, how to read a partial response body. Also I don´t understand why I´m getting this error code at all, since the content length sent by the server is correct and I was able to validate this with multiple Tools. Is it possible, that the beast client somehow closes the connection? And if so, what can I do about that? – abcd Feb 15 '21 at 12:06
  • @abod Nothing like that happens spontaneously. It will just be in the code as written. I can't see your code so I can't comment on that. And the way you read partial messages is obvious in the above, because some of the test cases _are_ partial messages, and it reports the message details. See also this example that uses the parser to read large responses block-wise and process them on the fly https://stackoverflow.com/questions/61787334/how-to-read-data-from-internet-using-muli-threading-with-connecting-only-once/61809655#61809655 – sehe Feb 15 '21 at 14:14