SWIG Python and C++ std::string null-termination problem

Question

Possible Duplicate:
How Python can get binary data(char*) from C++ by SWIG?

I have a SWIG based C++ interface that can called from Python. There is a function in it, that has one std::string argument. It looks like this:

WrapperGet(const std::string& key);

If it is called from Python with a string argument that contains NUL character (e.g. binary data), then the std::string is truncated at the NUL character, and that is my problem, because it makes impossible to handle binary data.

What makes it interesting is that other way works perfectly:

std::string WrapperResult();

The string returned here can contain binary data without truncation. Has anybody any idea what has to be done?

UPDATE: I debugged the SWIG generated code and it turned out that the error was in the wrapper code on the C++ size: it used the c_str() member function of the std::string to get the string value.

Thanks for everybody's ideas and time!

http://stackoverflow.com/questions/164168/how-do-you-construct-a-stdstring-with-an-embedded-null — Flexo, Feb 24 '11 at 18:26
The problem is not with std::string, but SWIG _with_ std::string. Do you have how to compile SWIG to achieve this? — agazso, Feb 24 '11 at 18:37
Isn't the problem that he shouldn't be using std::string to represent NUL data. I don't know anything about SWIG but if you construct the string by passing a character array, you will get the problem you are seeing. Maybe a vector would work better. — Lou, Feb 24 '11 at 18:59
@Lou, a vector _would_ work better, but technically std::string should be able to handle nulls for the most part. This sounds like a SWIG issue. — Assaf Lavie, Feb 24 '11 at 19:02
@Lou: `std::string` can handle NUL characters perfectly, if instead of passing only a pointer into a char array you pass also the size, it will read the whole contents regardless of possible internal NULs. The constructor you mentioned is just provided for compatibility with C strings. — David Rodríguez - dribeas, Feb 24 '11 at 20:07

score 3 · Accepted Answer · answered Feb 24 '11 at 21:36

I've had to deal with this in the past, and I just grabbed the std::string conversion template from SWIG and tailored it a bit to use PyString_FromStringAndSize when creating a Python string from a std::string and the std::string constructor that accepts a size argument when going the other way.

That was with a really old version of SWIG, though - I thought the builtin conversion template in newer versions had been updated to do that automatically. Is it possible the problem is on the C++ side? (e.g. as in Mark Tolonen's examples, where the first example is truncated due to the embedded NULL in the constructor call without a size argument).

Mark Tolonen · Answer 2 · 2011-02-25T08:01:41.447

It is probably how the strings are constructed. Consider:

string blah(const string& x)
{
    string y("abc\x00def");
    return x + y;
}

string blah2(const string& x)
{
    string y("abc\x00def",7);
    return x + y;
}

Calling these from Python:

>>> import x
>>> x.blah('abc\x00def')
'abc\x00defabc'
>>> x.blah2('abc\x00def')
'abc\x00defabc\x00xyz'

The constructor for std::string(const char*) stops at the NULL, but clearly SWIG can pass in and return strings with a NULL in them.

SWIG Python and C++ std::string null-termination problem

2 Answers2