Are there downsides to using std::string as a buffer?

Question

I have recently seen a colleague of mine using std::string as a buffer:

std::string receive_data(const Receiver& receiver) {
  std::string buff;
  int size = receiver.size();
  if (size > 0) {
    buff.resize(size);
    const char* dst_ptr = buff.data();
    const char* src_ptr = receiver.data();
    memcpy((char*) dst_ptr, src_ptr, size);
  }
  return buff;
}

I guess this guy wants to take advantage of auto destruction of the returned string so he needs not worry about freeing of the allocated buffer.

This looks a bit strange to me since according to cplusplus.com the data() method returns a const char* pointing to a buffer internally managed by the string:

const char* data() const noexcept;

Memcpy-ing to a const char pointer? AFAIK this does no harm as long as we know what we do, but have I missed something? Is this dangerous?

Answer 1

Don't use std::string as a buffer.

It is bad practice to use std::string as a buffer, for several reasons (listed in no particular order):

• std::string was not intended for use as a buffer; you would need to double-check the description of the class to make sure there are no "gotchas" which would prevent certain usage patterns (or make them trigger undefined behavior).
• As a concrete example: Before C++17, you can't even write through the pointer you get with data() - it's const Tchar *; so your code would cause undefined behavior. (But &(str[0]), &(str.front()), or &(*(str.begin())) would work.)
• Using std::strings for buffers is confusing to readers of your function's definition, who assume you would be using std::string for, well, strings. In other words, doing so breaks the Principle of Least Astonishment.
• Worse yet, it's confusing for whoever might use your function - they too may think what you're returning is a string, i.e. valid human-readable text.
• std::unique_ptr would be fine for your case, or even std::vector. In C++17, you can use std::byte for the element type, too. A more sophisticated option is a class with an SSO-like feature, e.g. Boost's small_vector (thank you, @gast128, for mentioning it).
• (Minor point:) libstdc++ had to change its ABI for std::string to conform to the C++11 standard, so in some cases (which by now are rather unlikely), you might run into some linkage or runtime issues that you wouldn't with a different type for your buffer.

_{Also, your code may make two instead of one heap allocations (implementation dependent): Once upon string construction and another when resize()ing. But that in itself is not really a reason to avoid std::string, since you can avoid the double allocation using the construction in @Jarod42's answer.}

Answer 2

You can completely avoid a manual memcpy by calling the appropriate constructor:

std::string receive_data(const Receiver& receiver) {
    return {receiver.data(), receiver.size()};
}

That even handles \0 in a string.

BTW, unless content is actually text, I would prefer std::vector<std::byte> (or equivalent).

Answer 3

Memcpy-ing to a const char pointer? AFAIK this does no harm as long as we know what we do, but is this good behavior and why?

The current code may have undefined behavior, depending on the C++ version. To avoid undefined behavior in C++14 and below take the address of the first element. It yields a non-const pointer:

buff.resize(size);
memcpy(&buff[0], &receiver[0], size);

---

I have recently seen a colleague of mine using std::string as a buffer...

That was somewhat common in older code, especially circa C++03. There are several benefits and downsides to using a string like that. Depending on what you are doing with the code, std::vector can be a bit anemic, and you sometimes used a string instead and accepted the extra overhead of char_traits.

For example, std::string is usually a faster container than std::vector on append, and you can't return std::vector from a function. (Or you could not do so in practice in C++98 because C++98 required the vector to be constructed in the function and copied out). Additionally, std::string allowed you to search with a richer assortment of member functions, like find_first_of and find_first_not_of. That was convenient when searching though arrays of bytes.

I think what you really want/need is SGI's Rope class, but it never made it into the STL. It looks like GCC's libstdc++ may provide it.

---

There a lengthy discussion about this being legal in C++14 and below:

const char* dst_ptr = buff.data();
const char* src_ptr = receiver.data();
memcpy((char*) dst_ptr, src_ptr, size);

I know for certain it is not safe in GCC. I once did something like this in some self tests and it resulted in a segfault:

std::string buff("A");
...

char* ptr = (char*)buff.data();
size_t len = buff.size();

ptr[0] ^= 1;  // tamper with byte
bool tampered = HMAC(key, ptr, len, mac);

GCC put the single byte 'A' in register AL. The high 3-bytes were garbage, so the 32-bit register was 0xXXXXXX41. When I dereferenced at ptr[0], GCC dereferenced a garbage address 0xXXXXXX41.

The two take-aways for me were, don't write half-ass self tests, and don't try to make data() a non-const pointer.

Answer 4

From C++17, data can return a non const char *.

Draft n4659 declares at [string.accessors]:

const charT* c_str() const noexcept;
const charT* data() const noexcept;
....
charT* data() noexcept;

Answer 5

The code is unnecessary, considering that

std::string receive_data(const Receiver& receiver) {
    std::string buff;
    int size = receiver.size();
    if (size > 0) {
        buff.assign(receiver.data(), size);
    }
    return buff;
}

will do exactly the same.

Answer 6

The big optimization opportunity I would investigate here is: Receiver appears to be some kind of container that supports .data() and .size(). If you can consume it, and pass it in as a rvalue reference Receiver&&, you might be able to use move semantics without making any copies at all! If it’s got an iterator interface, you could use those for range-based constructors or std::move() from <algorithm>.

In C++17 (as Serge Ballesta and others have mentioned), std::string::data() returns a pointer to non-const data. A std::string has been guaranteed to store all its data contiguously for years.

The code as written smells a bit, although it’s not really the programmer’s fault: those hacks were necessary at the time. Today, you should at least change the type of dst_ptr from const char* to char* and remove the cast in the first argument to memcpy(). You could also reserve() a number of bytes for the buffer and then use a STL function to move the data.

As others have mentioned, a std::vector or std::unique_ptr would be a more natural data structure to use here.

Answer 7

One downside is performance. The .resize method will default-initialize all the new byte locations to 0. That initialization is unnecessary if you're then going to overwrite the 0s with other data.

Answer 8

I do feel std::string is a legitimate contender for managing a "buffer"; whether or not it's the best choice depends on a few things...

Is your buffer content textual or binary in nature?

One major input to your decision should be whether the buffer content is textual in nature. It will be less potentially confusing to readers of your code if std::string is used for textual content.

char is not a good type for storing bytes. Keep in mind that the C++ Standard leaves it up to each implementation to decide whether char will be signed or unsigned, but for generic black-box handling of binary data (and sometimes even when passing characters to functions like std::toupper(int) that have undefined behaviour unless the argument is in range for unsigned char or is equal to EOF) you probably want unsigned data: why would you assume or imply that the first bit of each byte is a sign bit if it's opaque binary data?

Because of that, it's undeniably somewhat hackish to use std::string for "binary" data. You could use std::basic_string<std::byte>, but that's not what the question asks about, and you'd lose some inoperability benefits from using the ubiquitous std::string type.

Some potential benefits of using std::string

Firstly a few benefits:

• it sports the RAII semantics we all know and love

• most implementations feature short-string optimisation (SSO), which ensures that if the number of bytes is small enough to fit directly inside the string object, dynamic allocation/deallocation can be avoided (but there may be an extra branch each time the data is accessed)

  • this is perhaps more useful for passing around copies of data read or to be written, rather than for a buffer which should be pre-sized to accept a decent chunk of data if available (to improve throughput by handling more I/O at a time)

• there's a wealth of std::string member functions, and non-member functions designed to work well with std::strings (including e.g. cout << my_string): if your client code would find them useful to parse/manipulate/process the buffer content, then you're off to a flying start

• the API is very familiar to most C++ programmers

Mixed blessings

• being a familiar, ubiquitous type, the code you interact with may have specialisations to for std::string that better suit your use for buffered data, or those specialisations might be worse: do evaluate that

Concerned

As Waxrat observed, what is lacking API wise is the ability to grow the buffer efficiently, as resize() writes NULs/'\0's into the characters added which is pointless if you're about to "receive" values into that memory. This isn't relevant in the OPs code where a copy of received data is being made, and the size is already known.

Discussion

Addressing einpoklum's concern:

std::string was not intended for use as a buffer; you would need to double-check the description of the class to make sure there are no "gotchas" which would prevent certain usage patterns (or make them trigger undefined behavior).

While it's true that std::string wasn't originally intended for this, the rest is mainly FUD. The Standard has made concessions to this kind of usage with C++17's non-const member function char* data(), and string has always supported embedded zero bytes. Most advanced programmers know what's safe to do.

Alternatives

• static buffers (C char[N] array or std::array<char, N>) sized to some maximum message size, or ferrying slices of the data per call

• a manually allocated buffer with std::unique_ptr to automate destruction: leaves you to do fiddly resizing, and track the allocated vs in-use sizes yourself; more error-prone overall

• std::vector (possibly of std::byte for the element type; is widely understood to imply binary data, but the API is more restrictive and (for better or worse) it can't be expected to have anything equivalent to Short-String Optimisation.

• Boost's small_vector: perhaps, if SSO was the only thing holding you back from std::vector, and you're happy using boost.

• returning a functor that allows lazy access to the received data (providing you know it won't be deallocated or overwritten), deferring the choice of how it's stored by client code

Answer 9

use C++23's string::resize_and_overwrite

https://en.cppreference.com/w/cpp/string/basic_string/resize_and_overwrite

https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p1072r10.html

[[nodiscard]] static inline string formaterr (DWORD errcode) {
    string strerr;
    
    strerr.resize_and_overwrite(2048, [errcode](char* buf, size_t buflen) {
        // https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-formatmessage
        return FormatMessageA(
            FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
            nullptr,
            errcode,
            0,
            buf,
            static_cast<DWORD>(buflen),
            nullptr
        );
    });
    
    return strerr;
}