C++ STL-based sending of binary serialized data for network transmission using sockets without the use of libraries

Question

I need to send a number of complex objects over the network to a peer. I have written the code to serialize them using ostream and the operator<< for each class member in the objects that need to be serialized. The code I have written works successfully for serialization and network sending (htonl(), htons() etc. correctly used--I checked the aforementioned by writing to an ofstream (local file) in binary format (std::ios::bin). My next task, writing this binary data over the network socket, is where I am having issues.

I have Socket class which translates std::string objects into C-style strings before sending them over the socket like so:

int Socket::send (const std::string goodies) const
{
    status = ::send (socket, goodies.c_str(), goodies.size(), 0);
            return status;
}

the same Socket class, which I use in the receiver, uses the recv() to place the incoming message into a std::string before passing it to the deserializing application:

int Socket::recv (std::string& goodies)
{
    char buf [1024];
    goodies = "";
    memset (buf, 0, 1025);

    int status = ::recv (socket, buf, 1024, 0);

    if (status < 0)
    {
        return -1;
    }
    else if (status == 0)
    {
        return 0;
    }
    else
    {
        goodies = buf;
        return status;
    }
}

I do the sending using the following code:

ostringstream os (std::ios::binary);
GiantObject giantComplexObjectWithWholeLoadOfOtherObjects;
// Initialize and set up 
// giantComplexObjectWithWholeLoadOfOtherObjects.

// The following call works well--I tested it by writing to ofstream locally (file)
// and checked it out using a hex dump.
// Now, of course, my intent is to send it over the network, so I write to
// ostream&:
giantComplexObjectWithWholeLoadOfOtherObjects.serialize (os);

std::string someStr = os.str(); // Get me the stream in std::string format

mySocket.send(someStr); // Does not work--send sent correctly, but recv received 0 bytes

However, if I try:

std::string someStr ("Some silly string");
mySocket.send (someStr); // received by receiver (receiver is similar arch).

Thereby, something is not right about my call to send binary std::string to the socket. Any help is greatly appreciated. Once again, I do not want to use Boost, protobuf etc.

PS: I have spent a considerable amount of time looking over the old posts here, and the first response these types of questions receive is to use Boost. Please--there are other non-Boost, non-Protobuf ways, and I want to understand those other ways. I do appreciate what Boost and Protobuf bring to the table, I just want to do this in the manner I have set up the code.

This may be coming as a very late reply - but may help future visitors. But since I come from embedded background I was in your shoes looking for a homegrown solution until I stumbled upon https://code.google.com/p/nanopb/ . Its the most lightweight library I could find. They even have an example of network server in c. — enthusiasticgeek, Feb 17 '14 at 03:21

Dave S · Accepted Answer · 2011-10-14T16:29:07.270

I believe one problem is the use of std::string as your container. Especially the line goodies = buf;. The std::string assignment operator from a char* assumes that the input ends at the first NUL. Since binary data can contain arbritrary NUL characters, it will likely terminate earlier than you expect.

For an application such as this, I recommend the use of std::vector<char> or std::vector<unsigned char>. Since the memory in the buffer is always contiguous, you can access the underlying buffer by taking the address of the first element.

For the receive side, you will want to use std::copy to copy the data. Or you can set it up to use a vector buffer, and swap once the receive is complete. This assumes that you want goodies unmodified if the status is <= 0. If you can accept goodies being changed even in those cases, you can resize it prior to calling ::recv(), and avoid creating the extra vector.

int Socket::send (const std::vector<char>& goodies) const {
    status = ::send (socket, &goodies[0], goodies.size(), 0);            
    return status; 

int Socket::recv (std::vector<char>& goodies)  {    
  std::vector<char> buffer(1024);
  int status = ::recv (socket, &buf[0], buf.size());
  if (status < 0) 
  {
     return -1;      
  }
  else if (status == 0)
  { 
     return 0;
  }
  else
  {
     buffer.resize(status);
     goodies.swap(buffer);
     return status;      
  } 
}

Thanks for your replies. I guess I come from a very "C" background--I have much to learn about STL. I have a few ancillary questions-- (1) How does one specify an ostream of type binary (std::ios::binary works only for ostringstream), and does it matter when I am putting out a binary stream? (b) How does one move data from ostringstream to vector? And, in the manner of this post (http://stackoverflow.com/questions/4340396/does-the-c-standard-mandate-poor-performance-for-iostreams-or-am-i-just-deali), what are the overheads of doing these operations? — Sonny, Oct 16 '11 at 13:11
FYI, starting with C++11, it might be cleaner to say buf.data() instead of &buf[0]. Also, he could have used assign(), goodies.assign(buffer, status); — Rahly, Aug 19 '14 at 22:28

acm · Answer 2 · 2011-10-14T17:02:54.357

One issue in the above is that you are assuming that 'send' will always send goodies.size() bytes in one call. It is not required to do so: you need to call send repeatedly until you have sent goodies.size() bytes. In addition, you probably want to take goodies by const ref, rather than by value to avoid copying your huge object.

On the recv side, again, you have no assurances that recv will return to you a complete object. You will need to repeatedly consume data into a buffer until you have determined that you have extracted a complete object. This means that when you send your object, you will either need to use a message oriented protocol like SCTP that delivers message boundaries at the protocol level, or use a streaming protocol, but include framing information in your message format. That will allow the consumer to extract a header describing the length of data to expect in the object.

This, BTW, is why you will get lots of replies to just use an existing framework, because they take care of these sorts of framing and message assembly issues for you.

Also, I second Dave S's point below about NULL termination and his suggestion to not use std::string. — acm, Oct 14 '11 at 15:19

score 1 · Answer 3 · answered Oct 14 '11 at 15:10

1

One thing that pops up immediately is that you are doing both input and output with the std::ostringstream, which I believe is only meant to do output. Are you using void ostringstream::str(string)? Or are you using the extraction operator? I'm not sure if either would work, but I would think the former would work. However, if you're going to do input and output on the stringstream, you should really just use stringstream (which sets the mode to ios_base::in | ios_base::out.

answered Oct 14 '11 at 15:10

gred

612
1
8
15

Thanks for your comments--That was a typo. Upon futher spelunking, I think the main issue was the std::string and NULL character confusion. I am trying to rework everything using std::vector, but am still trying to figure the details out. – Sonny Oct 16 '11 at 13:20
ah yes, NULL characters... Boost has a static_vector<> which is fixed size. that might be more appropriate for your buffer. – gred Oct 17 '11 at 01:19

C++ STL-based sending of binary serialized data for network transmission using sockets without the use of libraries

3 Answers3