Handling large http response using boost::beast

Question

The following code use to get http response message:

  boost::beast::tcp_stream stream_;

  boost::beast::flat_buffer buffer;
  boost::beast::http::response<boost::beast::http::dynamic_body> res;
  boost::beast::http::read(stream_, buffer, res);

However, In some cases, based on the preceding request, I can expect that the response message body will include large binary file.

Therefore, I’d like to read it directly to the filesystem and not through buffer variable to avoid excessive use of process memory. How can it be done ?

in Objective-c framework NSUrlSession there's an easy way to do it using NSURLSessionDownloadTask instead of NSURLSessionDataTask, so I wonder if it's also exist in boost.

Thanks !

sehe · Accepted Answer · 2021-09-01T20:01:37.883

In general, you can use the http::buffer_body to handle arbitrarily large request/response messages.

If you specifically want to read/write from a filesystem file, you can have the http::file_body instead.

Full Demo `buffer_body`

The documentation sample for buffer_body is here https://www.boost.org/doc/libs/1_77_0/libs/beast/doc/html/beast/using_http/parser_stream_operations/incremental_read.html.

Using it to write to std::cout: Live On Coliru

#include <boost/asio.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/beast.hpp>
#include <boost/beast/websocket.hpp>
#include <iostream>

namespace net       = boost::asio;
namespace beast     = boost::beast;
namespace http      = beast::http;
using tcp           = net::ip::tcp;
using socket_t      = tcp::socket;

/*  This function reads a message using a fixed size buffer to hold
    portions of the body, and prints the body contents to a `std::ostream`.
*/

template<
    bool isRequest,
    class SyncReadStream,
    class DynamicBuffer>
void
read_and_print_body(
    std::ostream& os,
    SyncReadStream& stream,
    DynamicBuffer& buffer,
    beast::error_code& ec)
{
    http::parser<isRequest, http::buffer_body> p;
    http::read_header(stream, buffer, p, ec);
    if(ec)
        return;
    while(! p.is_done())
    {
        char buf[512];
        p.get().body().data = buf;
        p.get().body().size = sizeof(buf);
        http::read(stream, buffer, p, ec);
        if(ec == http::error::need_buffer)
            ec = {};
        if(ec)
            return;
        os.write(buf, sizeof(buf) - p.get().body().size);
    }
}

int main() {
    std::string host = "173.203.57.63"; // COLIRU 20210901
    auto const port  = "80";

    net::io_context ioc;
    tcp::resolver   resolver{ioc};

    socket_t s{ioc};
    net::connect(s, resolver.resolve(host, port));

    write(s, http::request<http::empty_body>{http::verb::get, "/", 11});

    beast::error_code  ec;
    beast::flat_buffer buf;

    read_and_print_body<false>(std::cout, s, buf, ec);
}

Full `file_body` example

This is much shorter, writing to body.html:

Live On Coliru

#include <boost/asio.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/beast.hpp>
#include <boost/beast/websocket.hpp>
#include <iostream>

namespace net       = boost::asio;
namespace beast     = boost::beast;
namespace http      = beast::http;
using tcp           = net::ip::tcp;
using socket_t      = tcp::socket;

int main() {
    std::string host = "173.203.57.63"; // COLIRU 20210901
    auto const port  = "80";

    net::io_context ioc;
    tcp::resolver   resolver{ioc};

    socket_t s{ioc};
    net::connect(s, resolver.resolve(host, port));

    write(s, http::request<http::empty_body>{http::verb::get, "/", 11});

    beast::error_code  ec;
    beast::flat_buffer buf;
    http::response<http::file_body> res;
    res.body().open("body.html", beast::file_mode::write_new, ec);
    if (!ec.failed())
    {
        read(s, buf, res, ec);
    }

    std::cout << "Wrote 'body.html' (" << ec.message() << ")\n";
    std::cout << "Headers " << res.base() << "\n";
}

Prints

Wrote 'body.html' (Success)
Headers HTTP/1.1 200 OK 
Content-Type: text/html;charset=utf-8
Content-Length: 8616
Server: WEBrick/1.4.2 (Ruby/2.5.1/2018-03-29) OpenSSL/1.0.2g
Date: Wed, 01 Sep 2021 19:52:20 GMT
Connection: Keep-Alive

With file body.html; wc body.html showing:

body.html: HTML document, ASCII text, with very long lines
 185  644 8616 body.html

Beyond: streaming to child processes and streaming processing

I have an advanced example of that here: How to read data from Internet using muli-threading with connecting only once?.

Thanks, that's working perfectly. I just wish there also an option to automatically direct the response to file without user interaction. — Zohar81, Sep 01 '21 at 12:49
You're actually right. I forgot aout `file_body`, mentally assuming it was only for serializing bodies from static files. I rewrote a [simple example of buffer_body to ostream](http://coliru.stacked-crooked.com/a/0dc099823130fcdb) to use [file_body instead](http://coliru.stacked-crooked.com/a/5b6be7759d066725) and it's much easier indeed. Adding that to the answer text. — sehe, Sep 01 '21 at 19:52
Much updated the answer. Thanks for making me notice my mistaken assumption :) I simply forgot I had seen that in the source code once in the past. — sehe, Sep 01 '21 at 20:02
Thanks for the detailed answer. However, for some reason, after i use the `file_mode` method, it somehow disconnected the socket, so i had to reconnect in order to re-use it. I haven't encountered such behavior when working with the chunk-base technic... Perhaps do you know why ? — Zohar81, Sep 02 '21 at 08:32
That's normal HTTP behaviour with `Connection: close` headers (assumed if absent). The alternative requires Content-Length and/or chunked encoding to be used (so you don't treat eof as end of response) — sehe, Sep 02 '21 at 10:30

Handling large http response using boost::beast

1 Answers1

Full Demo buffer_body

Full file_body example

Beyond: streaming to child processes and streaming processing

Full Demo `buffer_body`

Full `file_body` example