Boost asio synchronous https call- Json response have unintended character

Question

we are migrating from http to https boost asio sysnchornous call and I am using the below code to make https synchoronous call with ssl certificate validation and we got the response into multiple lines; as of now i have removed the line feed character(\r\n)(upstream system is saying that they are sending response in single line and without any extra character as described below) and tried to parse the response but sometimes we are getting the response with extra characters in key value pairs as shown below:

try{
fast_ostringstream oss;
boost::asio::streambuf request_;
boost::asio::streambuf response_;
boost::system::error_code ec;
boost::asio::ssl::context ctx(boost::asio::ssl::context::sslv23);
ctx.set_verify_mode(boost::asio::ssl::verify_peer);
ctx.set_default_verify_paths(ec);
if (ec)
{
        fast_ostringstream oss;
        oss << "Issue in settign the default path:" << ec.message();
        PBLOG_INFO(oss.str());
}
oss << ec.message();
ctx.add_verify_path("/home/test/pemcert/");
ctx.set_options(boost::asio::ssl::context::default_workarounds |
       boost::asio::ssl::context::no_sslv2 |
       boost::asio::ssl::context::no_sslv3);
boost::asio::ssl::stream<boost::asio::ip::tcp::socket> socket(io_service,ctx);

std::ostream request_stream(&request_);
request_stream << "POST " << server_endpoint << " HTTP/1.1\r\n";
request_stream << "Host: " << hostname << "\r\n";
request_stream << "Accept: */*\r\n";
request_stream << authorization_token << "\r\n";
request_stream << client_name << "\r\n";
request_stream << "Content-Length: " << req_str.length() << "\r\n";
request_stream << "Content-Type: application/x-www-form-urlencoded \r\n";
request_stream << "Connection: close\r\n\r\n";
request_stream << req_str << "\r\n";
tcp::resolver resolver(io_service);
tcp::resolver resolver(io_service);
tcp::resolver::query query(hostname, port_no);

tcp::resolver::iterator endpoint_iterator = resolver.resolve(query);
tcp::resolver::iterator end;

boost::system::error_code error  = boost::asio::error::host_not_found;
boost::asio::connect(socket.lowest_layer(), endpoint_iterator, error);
boost::system::error_code echs;
socket.handshake(boost::asio::ssl::stream_base::client, echs);
boost::asio::write(socket, request_);
PBLOG_INFO("Trac Request successfully sent");

// Read the response status line.
boost::asio::read_until(socket, response_, "\r\n");
string res=make_string(response_);

// Check that response is OK.
std::istream response_stream(&response_);
std::string http_version;
response_stream >> http_version;
unsigned int status_code;
response_stream >> status_code;
std::string status_message;
std::getline(response_stream, status_message);

if (!response_stream || http_version.substr(0, 5) != "HTTP/")
{
        PBLOG_WARN("Invalid response\n");
}
if (status_code != 200)
{
        fast_ostringstream oss;
        oss << "Response returned with status code: " << status_code << "\n";
        PBLOG_WARN(oss.str());
}
boost::asio::read(socket, response_, boost::asio::transfer_all(), error);
if (error.value() != 335544539 && strcmp(error.category().name(),"asio.ssl") != 0 )
{
    fast_ostringstream oss;
    oss << "Error : " << error.message() << "Value:" << error.value() << "Category Name:" << error.category().name();
    PBLOG_WARN(oss.str());
    return false;
}
else
{
     string message = make_string(response_);
     size_t pos = message.find( "header" );
     if( pos != std::string::npos)
     {
          pos = pos - 2;
          string msg = message.substr(pos, message.length());
          msg.erase(std::remove(msg.begin(),msg.end(),'\n'),msg.end());
          msg.erase(std::remove(msg.begin(),msg.end(),'\r'),msg.end());
          msg.erase(msg.size()-1); //to ignore the short read error
          response = msg;

     }
     else
     {
        fast_ostringstream oss;
          oss << "Invalid Response: " << message;
          PBLOG_WARN(oss.str());
          return false;
     }
     socket.lowest_layer().shutdown(tcp::socket::shutdown_both);
}
}

Json response:

I couldn't paste the full response due to security reason but small part where the extra character(here 21f0 is getting appended while we got the resposne)is getting added is shown below:

"SGEType":{"decisionKey"21f0:"SGMtype","decisionValue":null,"decisionGroup":"partyTranslations","ruleName":"Party Details Translations"}

Please let me know whether i am reading from the socket is accurate or needs modification.

Answer 1

I couldn't paste the full response due to security reason but small part where the extra character(here 21f0 is getting appended while we got the response)is getting added is shown below:

We have no way of knowing. How big is the response? My wild stab at things is that it might be using chunked encoding which you are potentially mis-handling because you manually "non-parse" HTTP responses?

In that case, my earlier answer might be kind of prophetic:

Luckily, you can also refer to that live example to see how to use Boost Beast to read the response correctly.

Live On Coliru

I'll also repeat the summary because the lesson is an important one:

---

Side note: Just reading until EOF would probably have worked for HTTP/1.0. But the server might rightfully reject that version or choose to respond with HTTP/1.1 anyways.