6

My starting point is to create a simple downloader code from the boost beast http_client_async example at boost http_client_async. In this scenario i want to write the received body into a file.

So I exchanged the string body into a file_body, to write the received data:

 http::response_parser<http::file_body> res_;

And simply rewrote the on_write method to

void on_write( boost::system::error_code ec,
               std::size_t bytes_transferred )
{
    boost::ignore_unused(bytes_transferred);

    if(ec)
        return fail(ec, "write");

    boost::system::error_code ec_file;
    res_.body().open("myTest.txt", boost::beast::file_mode::write, ec_file);

    // test for ec_file missing 

    // Receive the HTTP response
    http::async_read(socket_, buffer_, res_,
        std::bind(
            &session::on_read,
            shared_from_this(),
            std::placeholders::_1,
            std::placeholders::_2));
}

So but now, some of the received data bodies are to big:

read: body limit exceeded

and I try to increase the body limit.

In case of using a parser instead of a message, the size limit of the requested body can be changed with the body_limit() method.

Is there an easy way to increase the body size limit from a message as well?

  • 1
    to use the `response_parser` instead of `response` is the best way to go, I think. It got confused here. A response parser is not a message. When trying to get the body for calling open, res_ needs a call to `res_.get().body().open("myTest.txt", boost::beast::file_mode::write, ec_file);`. The response_parser `res_` is able to handle the member variable body_limit(). –  May 15 '18 at 12:18

1 Answers1

11

Beast's HTTP interfaces are grouped into layers. The first layer has the message-oriented stream algorithms, which operate on an HTTP message container. These are designed for simplicity, but allow for very little customization. The next layer is the serializer/parser oriented interface. This requires maintaining the lifetime of a serializer (for writing) or a parser (for reading) for the duration of the stream operation. It is a little more complex but correspondingly allows for more customization.

Adjusting the maximum size of the message body requires using the parser-oriented interface, as you have noted in your comment:

namespace http = boost::beast::http;
http::response_parser<http::file_body> parser;

// Allow for an unlimited body size
parser.body_limit((std::numeric_limits<std::uint64_t>::max)());
...
http::read(socket, buffer, parser);
Vinnie Falco
  • 5,173
  • 28
  • 43
  • 1
    Thanks for the clarification! I worked with the code for a while now. Reading the code + ducumentation API helps more than the documentation examples at the moment as the possibilities to use beast are really huge! –  Jun 04 '18 at 13:08
  • I think std::numeric_limits::max should be std::numeric_limits::max(). – bisarch Nov 22 '18 at 07:10
  • 2
    Yes, you are right! Also, the function name should be surrounded with parenthesis in case someone defines `max` as a macro (tsk tsk Windows.h) – Vinnie Falco Nov 22 '18 at 11:07
  • `http::response_parser` is not `move_insertable`. This prevents `std::vector::reserve`, `::resize` and more. How can I deal with this? All I really need is a usable vector of HTTP responses with no body limit. – Xeverous Jan 22 '20 at 23:57
  • @VinnieFalco I've basically gone through the same steps as the OP. It works, but I was wondering about using async_read_some as I'll be working a relatively resource constrained target. So far, I've failed and I get a "bad line ending" error on the 2nd read. I'm obviously doing something wrong; do I need to use async_read_some at all? If so, can you give me any pointers on where to look (do I need a serializer, or a custom parser?) Is there a better place to ask for help!? Many thanks. – cosimo193 Apr 24 '20 at 21:28
  • You need to inspect the bytes that you are receiving. Most likely, the other end is sending you invalid HTTP. – Vinnie Falco Apr 25 '20 at 14:13