boost::asio::streambuf retrieve xml data though https

Question

I'm struggling with streambuf management in Asio. I'm using boost 1.58 on ubuntu. First, here is the code:

#include <iostream>

#include <boost/bind.hpp>
#include <boost/asio.hpp>
#include <boost/asio/ssl.hpp>
#include <boost/asio/buffer.hpp>
#include <boost/asio/completion_condition.hpp>

class example
{
private:
    // asio components
    boost::asio::io_service service;
    boost::asio::ssl::context context;
    boost::asio::ip::tcp::resolver::query query;
    boost::asio::ip::tcp::resolver resolver;
    boost::asio::ssl::stream<boost::asio::ip::tcp::socket> socket;
    boost::asio::streambuf requestBuf, responseBuf;

    // callbacks
    void handle_resolve(const boost::system::error_code& err,
                            boost::asio::ip::tcp::resolver::iterator endpoint_iterator)
    {
        if (!err)
        {
            boost::asio::async_connect(socket.lowest_layer(), endpoint_iterator,
                boost::bind(&example::handle_connect, this,
                    boost::asio::placeholders::error));
        }
    }
    void handle_connect(const boost::system::error_code& err)
    {
        if (!err)
        {
          socket.async_handshake(boost::asio::ssl::stream_base::client,
              boost::bind(&example::handle_handshake, this,
                boost::asio::placeholders::error));
        }
    }
    void handle_handshake(const boost::system::error_code& err)
    {
        if (!err)
        {
            boost::asio::async_write(socket, requestBuf,
                boost::bind(&example::handle_write_request, this,
                    boost::asio::placeholders::error,
                    boost::asio::placeholders::bytes_transferred));
        }
    }

    void handle_write_request(const boost::system::error_code& err, size_t bytes_transferred)
        {
            if (!err)
            {
                boost::asio::async_read(socket, responseBuf,
                    boost::asio::transfer_at_least(1),
                    boost::bind(&example::handle_read, this,
                        boost::asio::placeholders::error,
                        boost::asio::placeholders::bytes_transferred));
            }
        }

    void handle_read(const boost::system::error_code& err,
                             size_t bytes_transferred)
    {
        if (!err)
        {
            boost::asio::async_read(socket, responseBuf,
                boost::asio::transfer_at_least(1),
                boost::bind(&example::handle_read, this,
                    boost::asio::placeholders::error,
                    boost::asio::placeholders::bytes_transferred));
        }
    }
public:
    example() : context(boost::asio::ssl::context::sslv23),
                resolver(service),
                socket(service, context),
                query("www.quandl.com", "443") {}

    void work()
    {
        // set security
        context.set_default_verify_paths();
        socket.set_verify_mode(boost::asio::ssl::verify_peer);

        // in case this no longer works, generate a new key from https://www.quandl.com/
        std::string api_key = "4jufXHL8S4XxyM6gzbA_";

        // build the query
        std::stringstream ss;

        ss << "api/v3/datasets/";
        ss << "RBA" << "/" << "FXRUKPS" << ".";
        ss << "xml" << "?sort_order=asc";
        ss << "?api_key=" << api_key;
        ss << "&start_date=" << "2000-01-01";
        ss << "&end_date=" << "2003-01-01";

        std::ostream request_stream(&requestBuf);
        request_stream << "GET /";
        request_stream << ss.str();
        request_stream << " HTTP/1.1\r\n";
        request_stream << "Host: " << "www.quandl.com" << "\r\n";
        request_stream << "Accept: */*\r\n";
        request_stream << "Connection: close\r\n\r\n";

        resolver.async_resolve(query,
            boost::bind(&example::handle_resolve, this,
                boost::asio::placeholders::error,
                boost::asio::placeholders::iterator));

        service.run();

        std::cout << &responseBuf;
    }
};

int main(int argc, char * argv[])
{
    // this is a test
    int retVal; try
    {
        example f; f.work();
        retVal = 0;

    }
    catch (std::exception & ex)
    {
        std::cout << "an error occured:" << ex.what() << std::endl;
        retVal = 1;
    }

    return retVal;

}

Here is my problem: the example works perfectly if the resulting data are not too long (a few thousands characters). However, as soon async_read returns an uneven number of characters (default bytes_transferred is 512 chars), the streambuf get corrupted and the next async_read call will contain a few extra characters.

I unsuccessfully tried many variations of the code above: using transfer_exactly(), calling streambuf.consume() to clear the buffer, passing another buffer as soon as I detect an uneven number of chars returned, etc. None of these solutions worked.

What am I missing here ? Thx

Answer 1

As determined in the comment exchange, the server is using chunked transfer encoding:

Chunked transfer encoding is a data transfer mechanism in version 1.1 of the Hypertext Transfer Protocol (HTTP) in which data is sent in a series of "chunks". It uses the Transfer-Encoding HTTP header in place of the Content-Length header, ...

Each chunk begins with a hexadecimal chunk length and a CRLF. If you are unfamiliar with chunked transfer, it will indeed appear that there are strange characters corrupting your data stream.

Chunked transfer encoding is generally used when it is not convenient to determine the exact length of the response body before it is sent. It follows that the receiver does not know the body length until processing the final zero-length chunk (note that trailing "headers", aka "trailers", may follow the final chunk).

With boost::asio, you can use async_read_until() to read the chunk header through the CRLF delimiter, parse the length, and then use async_read() with a transfer_exactly to get the chunk data. Note that once you begin using a streambuf for your reads, you should continue using the same streambuf instance because it may buffer additional data (extracting a particular amount of data from a streambuf is discussed here). Also note that chunk data ends with a CRLF (not included in the length) which you should discard.

It can be instructive (and even fun if you have the time and curiosity) to write your own HTTP client with boost::asio, but it isn't easy to cover all of the options (e.g. compression, trailers, redirection) in the HTTP standard. You may want to consider whether a mature client library like libcurl would suit your needs.